R: intercept may not be handled correctly

[tammandres]

Hi all,

I have enjoyed using the python package, but also needed the R package. I noticed that predicted probabilities from the EBM model created with R are too high when the dataset is imbalanced. I wonder if an intercept term may be missing from the model and/or from the predict_proba function? It looks like the ebm_predict_proba function simply adds together the contributions from each feature function, and applies the sigmoid function, but no intercept is added before applying the sigmoid.

Example: a dummy imbalanced dataset with 10% of positive cases. Average predicted probability is 0.39, which is much higher than 0.1.

df <- data.frame(x=seq(1, 100), y=c(rep(0, 90), rep(1, 10)))
clf <- ebm_classify(df['x'], df$y)
prob <- ebm_predict_proba(clf, df['x'])
print(mean(prob))
print(mean(df$y))

> print(mean(prob))
[1] 0.3949367
> print(mean(df$y))
[1] 0.1

Many thanks,
Andres

[paulbkoch]

Hi @tammandres -- I'll have to look into the imbalanced issue, but I'm pretty sure it isn't being caused by the lack of an intercept. EBM, like all GAMs, allow you to shift weight between features. To make the models identifiable, GAMs typically center each graph such that the average contribution of each graph is zero, and they move the weight to the intercept. This operation does not change the model's predictions. You can see this operation here in the python:

interpret/python/interpret-core/interpret/glassbox/ebm/utils.py

Line 281 in e1abdd1

intercept += score_mean

The R package does not currently center the graphs, but this should not lead to a change in the predictions.

[tammandres]

Hi @paulbkoch, many thanks for the response and the explanation! I am including the same code snippet in Python, which gives an expected result -- i.e. that the average predicted probability in an imbalanced dataset is roughly the proportion of positive cases:

import numpy as np
import pandas as pd
import interpret
from interpret.glassbox import ExplainableBoostingClassifier

df = pd.DataFrame(zip(np.arange(1, 101), np.append(np.zeros(90), np.ones(10))), 
                  columns=['x', 'y'])
clf = ExplainableBoostingClassifier()
clf.fit(df[['x']], df.y)
prob = clf.predict_proba(df[['x']])[:, 1]

print('Mean predicted prob: {}'.format(prob.mean()))
print('Mean observed prob: {}'.format(df.y.mean()))
print('Package version: {}'.format(interpret.__version__))

>Mean predicted prob: 0.10025197868529358
Mean observed prob: 0.1
Package version: 0.3.2

[DeepTensor-3070]

Hi I would be interested in investigating this issue if it is still open.

I will try to reproduce the discrepancy between the R and Python implementations compare the training and prediction pipelines (particularly how the prediction score is constructed) and identify where the behavior diverges. If I find the root cause, I would be happy to submit a fix along with tests.

Please let me know if that is okay.

[paulbkoch]

That would be great @DeepTensor-3070

[TemidayoA]

Hi @DeepTensor-3070 — just checking in on this, are you still actively working on it? Happy to help investigate or pick it up if you've moved on to other things, no worries either way. Wanted to check before duplicating effort.