Repository navigation
Conversation
Signed-off-by: andyluo7 <andy.luo@amd.com>
Collaborator
|
@andyluo7 this PR is labeled Could you please take a look and push an update to get CI green? Once the checks pass we can proceed with review/merge. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The domain-lock recovery test can fail after it has received the expected timeout report because it allows only five seconds for the spawned helper to exit. AMD #13103 at integration commit
edc0e5144fe7e6b6b8ee4bccdce8518b3efadbebreached that exit assertion withprocess.exitcode=None; its other 4,649 tests passed.Give the helper the same 60-second process-completion budget already used by the neighboring filesystem-store multiprocessing tests. Keep the tested 0.1-second lock deadline, five-second result-response bound, both required/preferred policies, typed retryable domain error, owner-lock protection, metadata preservation and fresh-runtime recovery assertions. Keep forced cleanup bounded to five seconds and explicitly fail if the child survives. Failure messages include the PID and exit status.
This test is byte-identical between main and the failing integration before this change. The log proves that all policy/lock assertions before the join passed; it does not identify why process exit exceeded five seconds. This change separates the exit allowance from the runtime coordination contract without changing production behavior.
Validation: all changed-file pre-commit checks, including mypy and marker validation, pass. An AST comparison verifies all 17 existing semantic/exit assertions are preserved and one cleanup assertion is added. A real spawned-process control returns its result, remains alive after a five-second join, then exits zero within the completion allowance; every child/pipe is collected. This local control tests process handling, not Torch or ROCm. Exact-head runtime evidence now confirms both parameters, including fresh-runtime recovery, pass in AMD source #13106 and CUDA source #16493. AMD integration #13107 READY and MERGE copies also pass both parameters; each completes 4,650 tests, 68 skips, three subtests, exit0/nonsoft in637.31/645.33s. Source AMD Simple Other passes4,568/68skips/646.09s; CUDA passes4,607/34skips/586.58s. This lane uploads no artifacts; qualification uses terminal logs plus authoritative job exit metadata.
Standalone AMD aggregate failures are separate unchanged-main AuK graph expectations (two failures) and Breeze graph/RNG expectations (five failures), addressed by existing #8342 and #8201 included in the integration. They do not arise in this changed test. Full source/platform checks and human approval remain; this PR stays draft until review evidence is complete.
Current runtime evidence
Both actual recovery-policy parameters pass on AMD #13106 Simple Other (4568 passed, 68 skipped, three subtests, 646.09s) and CUDA #16493 Simple Other (4607 passed, 34 skipped, three subtests, 586.58s), exit0/nonsoft. CUDA source completes 18/18. Both d926 integration READY/MERGE Simple Other copies execute and pass the repair (4650 passed/68 skipped each).
AMD #13106 completes 22/27 actual passes with four failures and one timeout. Terminal logs identify Ulysses invalid device ordinal/AuK replay on shard2, AuK replay on shard3 and Diffusion Model, Breeze graph-cache/RNG assertions in R2-01, and shard4 timeout at test_tiled_decode_matches_the_whole_decode. These match existing #8165/#8342/#8201 repairs included in the integration. Source aggregate red does not establish a new host-runtime regression. Current-main reconciliation, review and human approval remain separate gates.