enhance PyTorch easyblock to print individual failed tests#2983
Merged
branfosj merged 4 commits intoeasybuilders:developfrom Aug 24, 2023
Merged
enhance PyTorch easyblock to print individual failed tests#2983branfosj merged 4 commits intoeasybuilders:developfrom
branfosj merged 4 commits intoeasybuilders:developfrom
Conversation
This was referenced Aug 10, 2023
Contributor
Author
|
@branfosj This change has been tested now multiple times and improves things. I added the possible further enhancements to the OP as found in that other build. |
branfosj
requested changes
Aug 24, 2023
Member
branfosj
left a comment
There was a problem hiding this comment.
One minor change and then this is good to go. It has been well tested with various PyTorch PRs.
branfosj
approved these changes
Aug 24, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(created using
eb --new-pr)This enhances the output in the log file by printing not only the test suite of failed tests but also the individual failed tests. This should make it easier to determine if there is only a single test in a test suite that fails for everyone (or most) so we can disable that single test
Edit: As identified in easybuilders/easybuild-easyconfigs#18421 (comment) there are more issues with those possible solutions:
run_test.pyto not shard at all (any maybe disable the exit-on-first-failure)export BUILD_ENVIRONMENT=slow-gradcheckas a hack to disable the parallelization: https://github.com/pytorch/pytorch/blob/v1.13.1/test/run_test.py#L721 although that might get forgotten when the code changes (and it seems to have changed in PyTorch 2 already and was only introduced in 1.13)--save-xml) which might be the best option but requires quite some work.Edit: As for 4.:
It needs 2 Python packages (i.e. builddependencies):
lxmlandunittest-xml-reportingand a patch for PyTorch to propagate--save-xml.But then it has folders named after the tests with 1 or more xml files containing e.g.
<testsuites><testsuite name="pytest" errors="0" failures="0" skipped="127" tests="476" time="408.891" timestamp="2023-08-16T13:49:53.750990" hostname="taurusi8002"><testcase classname="TestJitCPU" name="test_jit_alias_remapping_abs_cpu_float32" time="0.063" file="test_ops_jit.py" />which does look helpful