Repository navigation
LightGmb exception with an error code -1 and an error message of 'bad allocation'. #3615
Copy link
Copy link
Closed
Labels
Priority:2Work that is important, but not critical for the releaseWork that is important, but not critical for the releasearea-TreesTree-based trainers, including FastTree and LightGBMTree-based trainers, including FastTree and LightGBMyellowmaintenance onlymaintenance only
Description
System information
- OS version/distro: Windows 10 64 bit
- .NET Version (eg., dotnet --info): .net core 3.0
Issue
-
What did you do?
Created a file with a label and a vector and trained against it. -
What happened?
Get an exception with an error code -1 and an error message of 'bad allocation'.
I can't find the error code in the source code so not sure what causes it. it happens after it releases about 30gb of memory. looks like right on
29.04.2019 20:20:01: [Source=LightGBMMulticlass; Loading data for LightGBM, Kind=Trace] Channel finished. Elapsed 01:52:28.7166971. -
What did you expect?
I'd expect this to work as it works with a smaller file.
Source code / logs
Loading the data
public static IDataView GetDataViewAsVector(MLContext mlContext, FileInfo trainingFile, long? maxRows = null)
{
var loader = mlContext.Data.CreateTextLoader(options: new TextLoader.Options()
{
Columns = new[] {
new TextLoader.Column(name:"Label", dataKind: DataKind.String, index: 0),
new TextLoader.Column(name:"Features",dataKind:DataKind.Single,minIndex:1,maxIndex:40731)
},
HasHeader = false,
Separators = new[] { '|' },
UseThreads = true,
MaxRows=maxRows
});
var dv = loader.Load(trainingFile.FullName);
return dv;
}
Making the model
var dataView = DataViewUtils.GetDataViewAsVector(mlContext, trainingFile, null);
dataset = mlContext.Data.TrainTestSplit(dataView, testFraction: 0.1);
var start = DateTime.Now;
(long? training, long? validating) = ((dataset.TrainSet.GetColumn<string>(Label).LongCount(), dataset.TestSet.GetColumn<string>(Label).LongCount()));
this.OnUpdate?.Invoke($"Start training LightGBM on {training.Value:N0} rows of data");
this.OnUpdate?.Invoke($"Testing LightGBM wil be on {validating.Value:N0} rows of data");
var options =new LightGbmMulticlassTrainer.Options {
LabelColumnName = KeyColumn,
FeatureColumnName = Features,
Silent= false,
Verbose=true,
NumberOfThreads=8
};
var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: KeyColumn, inputColumnName: Label)
// .Append(mlContext.Transforms.Concatenate(outputColumnName: Features, inputColumnNames: Mapper.GetFieldNames()))
.Append(mlContext.MulticlassClassification.Trainers.LightGbm(options))
.Append(mlContext.Transforms.CopyColumns(inputColumnName: KeyColumn, outputColumnName: nameof(PredictedResult.PredictedLabelIndex)))
;
// Train the model.
var model = pipeline.Fit(dataset.TrainSet);
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Priority:2Work that is important, but not critical for the releaseWork that is important, but not critical for the releasearea-TreesTree-based trainers, including FastTree and LightGBMTree-based trainers, including FastTree and LightGBMyellowmaintenance onlymaintenance only
Activity
PeterPann23 commentedon Apr 30, 2019
same error as posted in issue 3340. This time it seems to be persistent, perhaps size does matter...
sfilipi commentedon May 1, 2019
Looking at issue #3340, it points to issue #3290. Can we use the attached project to try to repro?
najeeb-kazmi commentedon May 8, 2019
@PeterPann23 does this still reproduce with ML.NET v1.0.0, which was release a couple days ago?
PeterPann23 commentedon May 9, 2019
In regards to bad allocation
I've not had the error in the new version however I am trying it on servers with ECC memory now. I can try and run it locally again and see it the error still appears on "normal" ram.
I'm guessing it has to do with memory size, hence my offer to provide you with the data-set. I'm sure could can generate a CSV file with random values as well and have no need for my help in this matter. But as I have not found the location of the error message I am not able to trace it being related to Error-correcting code memory I did run RAM test (Windows Memory Diagnostic/ MdSched.exe) on the development box over the weekend and reported no issues.
matouskozak commentedon Oct 23, 2026
Triage: This looks like possible similar symptoms to #6817. Since this no longer reproduced, I'll close this one.