Skip to content

LightGmb exception with an error code -1 and an error message of 'bad allocation'. #3615

Closed

Description

@PeterPann23

System information

  • OS version/distro: Windows 10 64 bit
  • .NET Version (eg., dotnet --info): .net core 3.0

Issue

  • What did you do?
    Created a file with a label and a vector and trained against it.

  • What happened?
    Get an exception with an error code -1 and an error message of 'bad allocation'.
    I can't find the error code in the source code so not sure what causes it. it happens after it releases about 30gb of memory. looks like right on
    29.04.2019 20:20:01: [Source=LightGBMMulticlass; Loading data for LightGBM, Kind=Trace] Channel finished. Elapsed 01:52:28.7166971.

  • What did you expect?
    I'd expect this to work as it works with a smaller file.

Crash.txt

Source code / logs

Loading the data

 public static IDataView GetDataViewAsVector(MLContext mlContext, FileInfo trainingFile, long? maxRows = null)
        {
            var loader = mlContext.Data.CreateTextLoader(options: new TextLoader.Options()
            {
                Columns = new[] {
                    new TextLoader.Column(name:"Label", dataKind: DataKind.String, index: 0),
                    new TextLoader.Column(name:"Features",dataKind:DataKind.Single,minIndex:1,maxIndex:40731)
                },
                HasHeader = false,
                Separators = new[] { '|' },
                UseThreads = true,
                MaxRows=maxRows
            });
            var dv = loader.Load(trainingFile.FullName);
            return dv;
        }

Making the model

var dataView = DataViewUtils.GetDataViewAsVector(mlContext, trainingFile, null);
dataset = mlContext.Data.TrainTestSplit(dataView, testFraction: 0.1);

var start = DateTime.Now;
(long? training, long? validating) = ((dataset.TrainSet.GetColumn<string>(Label).LongCount(), dataset.TestSet.GetColumn<string>(Label).LongCount()));

this.OnUpdate?.Invoke($"Start training LightGBM on {training.Value:N0} rows of data");
this.OnUpdate?.Invoke($"Testing LightGBM wil be on {validating.Value:N0} rows of data");        

var options =new LightGbmMulticlassTrainer.Options {
                    LabelColumnName   = KeyColumn,
                    FeatureColumnName = Features,
                    Silent= false,
                    Verbose=true,
                    NumberOfThreads=8
                };


var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: KeyColumn, inputColumnName: Label)
                // .Append(mlContext.Transforms.Concatenate(outputColumnName: Features, inputColumnNames: Mapper.GetFieldNames()))
                .Append(mlContext.MulticlassClassification.Trainers.LightGbm(options))                          
                .Append(mlContext.Transforms.CopyColumns(inputColumnName: KeyColumn, outputColumnName: nameof(PredictedResult.PredictedLabelIndex)))                         
                ;
           
// Train the model.
var model = pipeline.Fit(dataset.TrainSet);

Activity

  1. PeterPann23 commented on Apr 30, 2019

    @PeterPann23
    Author

    same error as posted in issue 3340. This time it seems to be persistent, perhaps size does matter...

  2. sfilipi commented on May 1, 2019

    @sfilipi
    Member

    Looking at issue #3340, it points to issue #3290. Can we use the attached project to try to repro?

  3. najeeb-kazmi commented on May 8, 2019

    @najeeb-kazmi
    Member

    @PeterPann23 does this still reproduce with ML.NET v1.0.0, which was release a couple days ago?

  4. PeterPann23 commented on May 9, 2019

    @PeterPann23
    Author

    In regards to bad allocation
    I've not had the error in the new version however I am trying it on servers with ECC memory now. I can try and run it locally again and see it the error still appears on "normal" ram.

    I'm guessing it has to do with memory size, hence my offer to provide you with the data-set. I'm sure could can generate a CSV file with random values as well and have no need for my help in this matter. But as I have not found the location of the error message I am not able to trace it being related to Error-correcting code memory I did run RAM test (Windows Memory Diagnostic/ MdSched.exe) on the development box over the weekend and reported no issues.

  5. added
    Priority:2Work that is important, but not critical for the release
    area-TreesTree-based trainers, including FastTree and LightGBM
    and removed on Sep 8, 2026
  6. matouskozak commented on Oct 23, 2026

    @matouskozak
    Member

    Triage: This looks like possible similar symptoms to #6817. Since this no longer reproduced, I'll close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Priority:2Work that is important, but not critical for the releasearea-TreesTree-based trainers, including FastTree and LightGBMyellowmaintenance only

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions