How to append estimator after a LightGBM trainer with early stopping

Version: 0.11

When using early stopping (which is fantastic) with LightGBM you have to use:

var trainer = ctx.MulticlassClassification.Trainers.LightGbm(options);
var model = trainer.Fit(trainEncodedDv, testEncodedDv);

However, I need to append other estimators after the trainer. Like:

trainer = trainer.Append(ctx.Transforms.Conversion.MapKeyToValue(DefaultColumnNames.PredictedLabel))
  .Append(ctx.Transforms.Conversion.MapKeyToValue((nameof(MlNetPrediction.OriginalLabelData), DefaultColumnNames.Label)));

Which of course I cannot do since then I cannot call the two-arity fit method anymore.

How can I solve this issue?

Btw: I think early stopping is absolutely crucial for any machine learning and IMO all iterative algorithms should support it.

perhaps use a callback in OnFit.

I don't think that will work. My problem is that I cannot specify any validation set during training (Fit).

There is a lot of API methods like Train/TrainContext etc around this whole concept. But they're all internal or private so I cannot access any of them.

E.g. this is an API scenario test but actual users have no chance of accessing any of this:

machinelearning/test/Microsoft.ML.Tests/Scenarios/Api/Estimators/TrainWithInitialPredictor.cs

Lines 22 to 43 in 0831865

    
           var ml = new MLContext(seed: 1); 
        
           var data = ml.Data.LoadFromTextFile<SentimentData>(GetDataPath(TestDatasets.Sentiment.trainFilename), hasHeader: true); 
        
           // Pipeline. 
        
           var pipeline = ml.Transforms.Text.FeaturizeText("Features", "SentimentText"); 
        
           // Train the pipeline, prepare train set. Since it will be scanned multiple times in the subsequent trainer, we cache the  
        
           // transformed data in memory. 
        
           var trainData = ml.Data.Cache(pipeline.Fit(data).Transform(data)); 
        
           // Train the first predictor. 
        
           var trainer = ml.BinaryClassification.Trainers.SdcaNonCalibrated( 
        
               new SdcaNonCalibratedBinaryTrainer.Options { NumberOfThreads = 1 }); 
        
           var firstModel = trainer.Fit(trainData); 
        
           // Train the second predictor on the same data. 
        
           var secondTrainer = ml.BinaryClassification.Trainers.AveragedPerceptron("Label","Features"); 
        
           var trainRoles = new RoleMappedData(trainData, label: "Label", feature: "Features"); 
        
           var finalModel = ((ITrainer)secondTrainer).Train(new TrainContext(trainRoles, initialPredictor: firstModel.Model));

If I could access ITrainer I think I could call that further down the line...

When you have a callback into the class you can provide parameters to the EventArgs class. the trainer that provides the callback can provide the options that are supported, hence one could have one for muticlass and another for regression or TensorFolow. the call back can be registered in the trainer's options class

Do you mean the static API? I've changed back to the dynamic API for several reasons. Do you have an example? I can't seem to get this done in the dynamic API.

It's something that will have to get implemented by the team

@rauhs

@rauhs can you train the pipeline, score your data, and create a second pipeline with KeyToValue, and give it as input the scored data?

Would two pipelines work for adding the other estimator?

I think he is thinking about
pipeline.WithOnFitDelegate, there he gets back a model. Looking at the code implementation in the extensions one could really use the available variables when passing them in OnFit using the classical EventArgs implementation.

public static Scalar<float> LightGbm(this RegressionCatalog.RegressionTrainers catalog,
            Scalar<float> label, Vector<float> features, Scalar<float> weights = null,
            int? numberOfLeaves = null,
            int? minimumExampleCountPerLeaf = null,
            double? learningRate = null,
            int numberOfIterations = Defaults.NumberOfIterations,
            Action<LightGbmRegressionModelParameters> onFit = null)
        {
            CheckUserValues(label, features, weights, numberOfLeaves, minimumExampleCountPerLeaf, learningRate, numberOfIterations, onFit);

            var rec = new TrainerEstimatorReconciler.Regression(
               (env, labelName, featuresName, weightsName) =>
               {
                   var trainer = new LightGbmRegressionTrainer(env, labelName, featuresName, weightsName, numberOfLeaves,
                       minimumExampleCountPerLeaf, learningRate, numberOfIterations);
                   if (onFit != null)
                       return trainer.WithOnFitDelegate(trans => onFit(trans.Model));
                   return trainer;
               }, label, features, weights);

            return rec.Score;
        }

can you train the pipeline, score your data, and create a second pipeline with KeyToValue, and give it as input the scored data?

That's a great idea which I hadn't thought of. That also works. Thanks! Though it's not exactly very elegant code to... Maybe ML.NET should make this simpler.

Yes, I had made comments that we need a hook back into this so one could use it for reporting and or early stopping. Simply state cancel and take the model time passed, or tat rec.score is [to] low/ high after Iteration [n]. we can count the iterations based on the callback invocation... having external control would allow managed learning, at the moment this is what I'm missing the most.

having external control would allow managed learning, at the moment this is what I'm missing the most.

100% agree with this. I honestly don't know how we're supposed to use any algorithm without early stopping. It's a HUGE waste of resources re-training hundreds of SDCA classifiers with the right iterations or regularization parameters to avoid overfitting/underfitting. Having access to the model on each iteration for any iterative algorithms is something I'd put at the top of the requirements.

I agree, the only option you have now is to let it iterate N times, see if it improves and start all over or if you are lucky, continue. there are some algo's that can continue learning.

@sfilipi

@sfilipi Will you update the loop in WrappedLightGbmTraining (Microsoft.ML.Trainers.LightGbm.WrappedLightGbmTraining).

You could invoke an event there where you pass a EventArgs that can be used to break the loop and pass a reference to the best model as well as the improvement.

We can subscribe to that event and manage/log the duration, iteration count, etc on our side and could optionally ignore/ duplicate the logic in

if (iter - bestIter >= earlyStoppingRound)
{
    ch.Info($"Met early stopping, best iteration: {bestIter + 1}, best score: {bestScore / factorToSmallerBetter}");
    break;
}

As we could do this in production code on any iteration if you provide a EventArg with a "Stop" property.

I can imagine that other training classes would be able to use this as well

@PeterPann23

Thanks for the feedback @PeterPann23, it helps to know what users need.
As we get into planning 1.1 I think we'll take a look at all requests and prioritize.

Triage: From reading the discussion, it looks like the key issue has been addressed. In case you hit other issue or have suggestions, please open a new issue.

Navigation Menu

Repository navigation

How to append estimator after a LightGBM trainer with early stopping #3603

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	var ml = new MLContext(seed: 1);

	var data = ml.Data.LoadFromTextFile<SentimentData>(GetDataPath(TestDatasets.Sentiment.trainFilename), hasHeader: true);

	// Pipeline.
	var pipeline = ml.Transforms.Text.FeaturizeText("Features", "SentimentText");

	// Train the pipeline, prepare train set. Since it will be scanned multiple times in the subsequent trainer, we cache the
	// transformed data in memory.
	var trainData = ml.Data.Cache(pipeline.Fit(data).Transform(data));

	// Train the first predictor.
	var trainer = ml.BinaryClassification.Trainers.SdcaNonCalibrated(
	new SdcaNonCalibratedBinaryTrainer.Options { NumberOfThreads = 1 });

	var firstModel = trainer.Fit(trainData);

	// Train the second predictor on the same data.
	var secondTrainer = ml.BinaryClassification.Trainers.AveragedPerceptron("Label","Features");

	var trainRoles = new RoleMappedData(trainData, label: "Label", feature: "Features");
	var finalModel = ((ITrainer)secondTrainer).Train(new TrainContext(trainRoles, initialPredictor: firstModel.Model));

How to append estimator after a LightGBM trainer with early stopping #3603

Description

Activity

PeterPann23 commented on Apr 29, 2019

rauhs commented on Apr 30, 2019

PeterPann23 commented on Apr 30, 2019

rauhs commented on Apr 30, 2019

PeterPann23 commented on Apr 30, 2019

sfilipi commented on Apr 30, 2019

PeterPann23 commented on Apr 30, 2019

rauhs commented on May 2, 2019

PeterPann23 commented on May 2, 2019

rauhs commented on May 2, 2019

PeterPann23 commented on May 3, 2019

PeterPann23 commented on May 9, 2019

sfilipi commented on May 9, 2019

matouskozak commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions