Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test_data to ICounterfactualData #1892

Merged
merged 3 commits into from
Jan 7, 2023

Conversation

gaugup
Copy link
Contributor

@gaugup gaugup commented Jan 7, 2023

Description

Add test_data to ICounterfactualData so that UI can make use of the test_data returned by SDK backend for counterfactual table. This is related to the PR:- #1885.

Checklist

  • I have added screenshots above for all UI changes.
  • I have added e2e tests for all UI changes.
  • Documentation was updated if it was needed.

@codecov-commenter
Copy link

codecov-commenter commented Jan 7, 2023

Codecov Report

Merging #1892 (45139fa) into main (10d432f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1892   +/-   ##
=======================================
  Coverage   93.35%   93.35%           
=======================================
  Files          93       93           
  Lines        4574     4574           
=======================================
  Hits         4270     4270           
  Misses        304      304           
Flag Coverage Δ
unittests 93.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@romanlutz
Copy link
Contributor

What exactly is test_data here? Is that the test dataset? If so, are we duplicating it? I might be missing the larger point as this is distributed over several PRs, but passing around an entire dataset (additionally to what we're already passing around it seems?) worries me. A pointer to a centralized design document that explains the big picture while referencing these individual pieces of work may also help.

@gaugup
Copy link
Contributor Author

gaugup commented Jan 7, 2023

What exactly is test_data here? Is that the test dataset? If so, are we duplicating it? I might be missing the larger point as this is distributed over several PRs, but passing around an entire dataset (additionally to what we're already passing around it seems?) worries me. A pointer to a centralized design document that explains the big picture while referencing these individual pieces of work may also help.

For large data scenario, not all the test samples will be present in UI. For local counterfactuals we would need to respond with the test dataset row. The size of the test data is not a concern as there is only one point returned for local counterfactuals. For initial run of rai, we limit the number of test samples to 5k. Please bear in mind that the generated counterfactuals per point determine the size of the output and not the test data itself.

We discovered this late for local counterfactual work that @vinuthakaranth is driving. The design doc for large is already shared with the PM and dev teams to refer to.

@gaugup gaugup enabled auto-merge (squash) January 7, 2023 16:44
@gaugup gaugup merged commit d9821c4 into main Jan 7, 2023
@gaugup gaugup deleted the gaugup/AddTestDataPropertyCounterfactualData branch January 7, 2023 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants