New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logic to create feature importance is not using the correct model #146

Open

sharathmalladi opened this issue Aug 19, 2019 · 1 comment

Assignees

Contributor

sharathmalladi commented Aug 19, 2019

The feature importance should not use a model file that is prior to the policy reset date.

marco-rossi29 assigned sharathmalladi

Contributor Author

sharathmalladi commented Sep 5, 2019

Sharing more context on the current logic:
When a counterfactual evaluation is created with a start and end date, we currently use the model file that was saved on the start date as input to vw to run the feature importance:
https://github.com/microsoft/mwt-ds/blob/master/DataScience/ExperimentationAzure.py#L168

This is not the right thing if the model or policy was reset in between the evaluation period and the logic needs to be tweaked to adjust for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment