Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bootstrapper wrapper not being reset correctly #2574

Merged
merged 5 commits into from
May 31, 2024

Conversation

SkafteNicki
Copy link
Member

@SkafteNicki SkafteNicki commented May 30, 2024

What does this PR do?

Fixes #2556

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?
PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃


📚 Documentation preview 📚: https://torchmetrics--2574.org.readthedocs.build/en/2574/

@SkafteNicki SkafteNicki added the bug / fix Something isn't working label May 30, 2024
@SkafteNicki SkafteNicki added this to the v1.3.x milestone May 30, 2024
Copy link

codecov bot commented May 30, 2024

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 34%. Comparing base (744905c) to head (ae6e6f2).
Report is 97 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff            @@
##           master   #2574     +/-   ##
========================================
- Coverage      69%     34%    -34%     
========================================
  Files         314     314             
  Lines       17669   17673      +4     
========================================
- Hits        12185    6096   -6089     
- Misses       5484   11577   +6093     

@mergify mergify bot added the ready label May 30, 2024
@Borda
Copy link
Member

Borda commented May 31, 2024

@SkafteNicki mind checking:

Failed example:
    from torchmetrics.classification import MulticlassAccuracy
    from torchmetrics.wrappers import BootStrapper
    # creating metrics
    wrapped_metric = BootStrapper(MulticlassAccuracy(num_classes=3))
    # sample prediction and GT
    target = torch.tensor([0, 2, 0, 2, 0, 1, 0, 2])
    preds = torch.tensor([2, 1, 2, 0, 1, 2, 2, 2])
    # showing the metric results
    print(wrapped_metric(preds, target))  # this returns a dict with label info
Expected:
    {'mean': tensor(0.1476), 'std': tensor(0.0613)}
Got:
    {'mean': tensor(0.1333), 'std': tensor(0.1554)}

@SkafteNicki
Copy link
Member Author

@SkafteNicki mind checking:

Failed example:
    from torchmetrics.classification import MulticlassAccuracy
    from torchmetrics.wrappers import BootStrapper
    # creating metrics
    wrapped_metric = BootStrapper(MulticlassAccuracy(num_classes=3))
    # sample prediction and GT
    target = torch.tensor([0, 2, 0, 2, 0, 1, 0, 2])
    preds = torch.tensor([2, 1, 2, 0, 1, 2, 2, 2])
    # showing the metric results
    print(wrapped_metric(preds, target))  # this returns a dict with label info
Expected:
    {'mean': tensor(0.1476), 'std': tensor(0.0613)}
Got:
    {'mean': tensor(0.1333), 'std': tensor(0.1554)}

I assume it is due to the randomness in the bootstrapper. I try setting a seed in the docs and update the values

@SkafteNicki SkafteNicki requested a review from lantiga as a code owner May 31, 2024 10:00
@github-actions github-actions bot added the documentation Improvements or additions to documentation label May 31, 2024
@SkafteNicki SkafteNicki merged commit cce449b into master May 31, 2024
60 of 61 checks passed
@SkafteNicki SkafteNicki deleted the bugfix/bootstrapper branch May 31, 2024 10:21
@Borda
Copy link
Member

Borda commented May 31, 2024

I see, so we can seed the example...

Borda pushed a commit that referenced this pull request Aug 2, 2024
* fix + add to tests

* changelog

* update values

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit cce449b)
Borda pushed a commit that referenced this pull request Aug 2, 2024
* fix + add to tests

* changelog

* update values

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit cce449b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working documentation Improvements or additions to documentation ready
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BootStrapper.reset() does not reset properly
3 participants