Custom eval function in 1.1.0 receiving preds as sigmoid not raw #6464
Description
Hi
I believe XGBoost 1.2.0 is returning raw values in preds in feval where as XGBoost 1.1.0 used to return sigmoid preds
I have a model which has a custom eval function like this
`
def custom_feval(preds, dtrain):
if np.array(preds).ndim == 2:
preds = preds[:, 1]
labels = dtrain.get_label()
tpr_at_fpr = some_function(labels, preds)
auc = roc_auc_score(labels, preds)
logloss = log_loss(labels, preds, eps=1e-7)
return [('auc', auc), ('tpr_at_fpr', tpr_at_fpr),('logloss', logloss)]`
With XGBoost 1.1.0
This is what I get for a dataset
[0] train-error:0.03967 val-error:0.02159 train-auc:0.93982 train-tpr_at_fpr:0.70061 train-logloss:0.61084 val-auc:0.94130 val-tpr_at_fpr:0.67254 val-logloss:0.60692
Multiple eval metrics have been passed: 'val-logloss' will be used for early stopping.
Will train until val-logloss hasn't improved in 5 rounds.
[5] train-error:0.03311 val-error:0.01673 train-auc:0.96514 train-tpr_at_fpr:0.79165 train-logloss:0.36151 val-auc:0.96271 val-tpr_at_fpr:0.75977 val-logloss:0.34571
[10] train-error:0.03064 val-error:0.01589 train-auc:0.97087 train-tpr_at_fpr:0.81537 train-logloss:0.24024 val-auc:0.96195 val-tpr_at_fpr:0.77934 val-logloss:0.22235
[15] train-error:0.02840 val-error:0.01535 train-auc:0.97475 train-tpr_at_fpr:0.82979 train-logloss:0.17273 val-auc:0.96617 val-tpr_at_fpr:0.79503 val-logloss:0.15236
[20] train-error:0.02725 val-error:0.01510 train-auc:0.97767 train-tpr_at_fpr:0.84399 train-logloss:0.13428 val-auc:0.97040 val-tpr_at_fpr:0.80252 val-logloss:0.11266
[25] train-error:0.02647 val-error:0.01476 train-auc:0.98001 train-tpr_at_fpr:0.85747 train-logloss:0.11100 val-auc:0.97313 val-tpr_at_fpr:0.81216 val-logloss:0.08933
[30] train-error:0.02556 val-error:0.01452 train-auc:0.98246 train-tpr_at_fpr:0.86745 train-logloss:0.09638 val-auc:0.97582 val-tpr_at_fpr:0.81958 val-logloss:0.07551
[35] train-error:0.02484 val-error:0.01439 train-auc:0.98456 train-tpr_at_fpr:0.87771 train-logloss:0.08691 val-auc:0.97654 val-tpr_at_fpr:0.82317 val-logloss:0.06770
[40] train-error:0.02422 val-error:0.01419 train-auc:0.98628 train-tpr_at_fpr:0.88415 train-logloss:0.08011 val-auc:0.97766 val-tpr_at_fpr:0.82857 val-logloss:0.06221
[45] train-error:0.02335 val-error:0.01404 train-auc:0.98772 train-tpr_at_fpr:0.89376 train-logloss:0.07479 val-auc:0.97805 val-tpr_at_fpr:0.83555 val-logloss:0.05915
[49] train-error:0.02267 val-error:0.01394 train-auc:0.98872 train-tpr_at_fpr:0.89915 train-logloss:0.07154 val-auc:0.97859 val-tpr_at_fpr:0.83901 val-logloss:0.05743
So as you can see logloss is decreasing as below every 5 rounds
0.60692
0.34571
0.22235
0.15236
0.11266
and so on
With XGBoost 1.2.0 - same code, same dataset, same every setting and I get this
[0] train-error:0.03967 val-error:0.02159 train-auc:0.93982 train-tpr_at_fpr:0.70061 train-logloss:0.60822 val-auc:0.94130 val-tpr_at_fpr:0.67254 val-logloss:0.19816
Multiple eval metrics have been passed: 'val-logloss' will be used for early stopping.
Will train until val-logloss hasn't improved in 5 rounds.
[5] train-error:0.03311 val-error:0.01673 train-auc:0.96514 train-tpr_at_fpr:0.79165 train-logloss:0.50761 val-auc:0.96271 val-tpr_at_fpr:0.75977 val-logloss:0.17106
[10] train-error:0.03064 val-error:0.01589 train-auc:0.97087 train-tpr_at_fpr:0.81537 train-logloss:0.46209 val-auc:0.96195 val-tpr_at_fpr:0.77934 val-logloss:0.17408
Stopping. Best iteration:
[5] train-error:0.03311 val-error:0.01673 train-auc:0.96514 train-tpr_at_fpr:0.79165 train-logloss:0.50761 val-auc:0.96271 val-tpr_at_fpr:0.75977 val-logloss:0.17106
I have an early stop of 5 iterations. As you can see the log loss values are completely different and since it does not decrease - the training stops.
This has cause huge difference in production pipelines - took me quite a white to figure out.
If I change the code above to
`
def custom_feval(preds, dtrain):
if np.array(preds).ndim == 2:
preds = preds[:, 1]
labels = dtrain.get_label()
preds = 1.0 / (1.0 + np.exp(-preds))
tpr_at_fpr = some_function(labels, preds)
auc = roc_auc_score(labels, preds)
logloss = log_loss(labels, preds, eps=1e-7)
return [('auc', auc), ('tpr_at_fpr', tpr_at_fpr),('logloss', logloss)]`
Then 1.2.0 behaves the same as 1.1.1
As debug information
Without the sigmoid conversion (default 1.2.0) - some pred values in custom_feval are like
[-0.16546123 -0.16546123 -0.19932126 -0.19191627 -0.1952204 -0.18206893
0.16579191 -0.16546123 -0.16546123 -0.19731039]
With sigmoid they become like
[0.4587288 0.4587288 0.45033404 0.45216766 0.4513493 0.45460805
0.5413533 0.4587288 0.4587288 0.45083183]