You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This code is totally incorrect if someone is using a custom evaluation metric.
I only found out about this bug after wasting several hours tuning and the best parameters that were returned being obviously nonsense.
I then tried explicitly creating a study with direction set to 'maximize' and at least hit a warning message.
### Steps to reproduce
1. Setup a study to maximize the optimization
2. Create a custom `feval` callback function
3. In the params explicitly set the metric to something other than ("auc", "auc_mu", "ndcg", "map", "average_precision")
4. In the call to optuna_integration.lightgbm.train, set the `feval` callable to the function created in step 2
```python
def score_cb(preds, eval_data):
score = calculate_score(preds, eval_data.label)
return 'score', score, True
lgb_study = optuna.create_study(direction="maximize", study_name="LightGBM Auto Tune")
params = {
"objective": "regression",
"metric": "correlation",
"boosting_type": "gbdt",
}
model = opt_lgb.train(params, dtrain, study=lgb_study, num_boost_round=2000, valid_sets=[dtrain, dval], valid_names=["training", "validation"], feval=score_cb, callbacks=[opt_lgb.early_stopping(stopping_rounds=30), lgb.log_evaluation(1)])
Additional context (optional)
No response
The text was updated successfully, but these errors were encountered:
Expected behavior
If the eval metric has is_higher_better set to True then the objective should be maximized.
Environment
Error messages, stack traces, or logs
This code is totally incorrect if someone is using a custom evaluation metric.
I only found out about this bug after wasting several hours tuning and the best parameters that were returned being obviously nonsense.
I then tried explicitly creating a study with direction set to 'maximize' and at least hit a warning message.
Additional context (optional)
No response
The text was updated successfully, but these errors were encountered: