Sklearn kwargs

[gaw89]

This allows users of the Sklearn API to provide all Booster params via a **kwargs dict.

[RAMitchell]

Can you please document what happens if a parameter is provided in kwargs and as normal parameter? Does one override the other?

[gaw89]

@RAMitchell great idea. Adding that clarification now. As of now, kwargs will override normal parameters. Is this acceptable behavior?

Scratch that, it actually throws a TypeError because it's being provided multiple values for the same argument. I think this is actually the proper behavior. I am making adjustments accordingly.

I am also adding tests to demonstrate this behavior.

[wxchan]

It might not be recommended to use kwargs for sklearn. see the notes on this page: http://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.html.

[khotilov]

@gaw89 Could you please find out whether the requirement that @wxchan has pointed out is about consistency, clarity and safety (e.g., protecting against typos) required for an estimator to become an official part of sklearn (which is not a goal for xgboost), or whether using kwargs might actually break something.

[gaw89]

@khotilov, yes, trying to sort that out now. Unfortunately, the page that @wxchan linked doesn't give much info, so I might have to go through some Sklearn source... I may try to put in issue in for Sklearn as well to get clarification.

In the meantime, I am using this branch on my machine and it's working great. Allows me to use GPU in GridSearchCV! :)

[gaw89]

@khotilov and @wxchan, I filed an issue with Sklearn. Here's their response:

"It's the API of scikit-learn. Things are not garantied to work if you do
it differently. We don't support it."

So, it seems that it is primarily an API/support issue - they won't guarantee that things will not break if we use this. However, it does not guarantee that things will break either. From looking at the Sklearn source as it is presently, I don't see how this would cause problems, but I am not terribly familiar with the inner workings of Sklearn.

I'm not sure what the best course of action is here. It seems fairly unlikely that it will make things break, but there is the risk. At the same time, individually adding the various booster params to the init could be cumbersome both for maintenance and usage.

I am happy to work on updating these things individually if that is the decision.

[RAMitchell]

It seems likely that they want each parameter explicit in the init function for reflection purposes.

I think we go ahead with the documentation that this is an unsupported sklearn parameter and parameters passed this way are not gauranteed to interact correctly with sklearn.

It is clear that we need a way to pass a variety of parameters to the underlying xgboost model.

Great work @gaw89.

[gaw89]

@RAMitchell, thanks! I went ahead and added a note to indicate this caveat with usage of kwargs. Then I fixed my lint errors again... Sick of Spyder adding whitespace at the end of my lines.

[terrytangyuan]

Thank you!