Repository navigation
fix(sweeper): bound worker cleanup before pool replacement - #170
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
Included review availability: Your plan provides up to 12 included reviews per hour; 2 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (20)
🧰 Additional context used📓 Path-based instructions (5)Check unified CLI, Replay, Sweeper, and orchestration behavior together.⚙️ CodeRabbit configuration file Files:
Require coverage of the changed behavior and its negative or boundary cases.⚙️ CodeRabbit configuration file Files:
Check commands, defaults, supported runtimes, public names, and claims against executable behavior.⚙️ CodeRabbit configuration file Files:
Read REVIEW.md before commenting.⚙️ CodeRabbit configuration file Files:
Before making any change under: `python/aisimulate/src/aiconfigurator/generator/**` MUST read: `python/aisimulate/.claude/rules/generator-development.md` Before making any change under `python/aisimulate/collector/**` MUST read: `python/ais...📄 CodeRabbit inference engine (AGENTS.md) Files:
🪛 LanguageTooldocs/local-resources.md[style] ~146-~146: Consider using “who” when you are referring to people instead of objects. (THAT_WHO) 🔀 Multi-repo context ai-dynamo/dynamo, ai-dynamo/aiconfiguratorLinked repositories findingsai-dynamo/dynamo
ai-dynamo/aiconfigurator
🔇 Additional comments (4)
📝 SummaryRisk: MediumHuman attention should focus on:
Changed behavior
Areas for review
Evidence
WalkthroughPool cleanup now uses shared supervision helpers. Timed-out pools are reaped before replacement, normal cleanup allows bounded finalization, and regression tests cover worker termination, result preservation, and hung finalizers. ChangesBounded process-pool cleanup
Priority: ⬇️ Low Estimated code review effort: 3 (Moderate) | ~20 minutes Unblocks: 1 PR Merge Risk: ⚪ Minimal · up to The bounded Sweeper cleanup change has no identified current behavior that would block merging. 🚥 Pre-merge checks | ✅ 8✅ Passed checks (8 passed)
Comment |
|
The requested overlap audit compared #170 at #169 now includes the CLI/SDK supervisor, owned-process memory monitoring, runtime thread limits, deadlines, checkpoint evidence, launchers, and the supervision regression tests. Carrying the old #170 implementation forward would duplicate those changes and conflict with the newer candidate recovery and output contracts. The remaining concrete gap is direct low-level Sweeper process-pool cleanup: the non-resource-aware path terminates workers without waiting before replacement and waits without a bound during normal shutdown. I will rebase this draft onto #169 and retain only reuse of #169's existing The PR will remain a draft. #174's integrated workload qualification remains separate. |
|
The new real-worker shutdown regression exposed a concrete gap inherited from #169's cleanup helper at [P1] Do not call executor shutdown again before escalating stuck workers. Keep the existing executor manager as the sole worker-join owner, signal workers, wait only for the bounded grace period, kill survivors, and finish executor shutdown after the manager exits. This belongs in the narrowed cleanup follow-up and will be covered together with the direct Sweeper lifecycle tests. |
156ab07 to
0a67422
Compare
|
Updated in The PR now has one commit and four changed files. It retains only bounded direct Sweeper pool cleanup, the shared helper's shutdown-lock correction, real-worker regressions, and the documentation for that behavior. The duplicated CLI/SDK supervisor, resource policies, launchers, and result changes are supplied by #169. The cleanup finding above is fixed: shutdown is completed after the manager exits, so a hung finalizer cannot block SIGKILL escalation. The complete Sweeper/resource/scheduler/supervision/unified CLI run passed 721 tests, with 5 expected optional Dynamo integration skips. The replacement regression also fails against the original #169 implementation and passes with the fix. Configured lint/format, docs links, ownership coverage, and whitespace checks pass. Final Codex self-review found no remaining actionable issue in this exact four-file diff. The draft remains based on #169. Current-head Fast CI passed; Full CI and human review have not been completed for this head. #174 needs its qualification branch refreshed separately against the narrowed stack. |
|
@coderabbitai full review Please review exact head The complete PR diff is four files: direct Sweeper pool cleanup now uses the existing bounded cleanup helpers; the shared termination helper avoids a shutdown-lock deadlock before killing stuck workers; real spawned-worker regressions cover termination resistance, normal finalizers, hung finalizers, and preserved completed results; documentation describes the exact scope. Codex re-read the full current diff and cleanup/error paths and found no remaining actionable issue. The unchanged application/test code passed 721 local tests with five expected optional Dynamo integration skips before the documentation-only parent update. Focused current-head regression validation is running. Please identify any remaining actionable findings against this exact head; current-head Fast CI, this review, and the Codex review must complete before Full CI dispatch. Human/CODEOWNER approval remains separate. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '=== checked-out revision ==='
git rev-parse HEAD
git status --short
printf '%s\n' '=== exact stacked diff file list ==='
git diff --name-status fda69bd4bce4dace3a43566ef6b25e51c8c13e65 c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89
printf '%s\n' '=== exact stacked diff ==='
git diff --find-renames --find-copies --no-ext-diff --unified=80 \
fda69bd4bce4dace3a43566ef6b25e51c8c13e65 \
c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89Length of output: 980 🏁 Script executed: #!/bin/bash
set -u
echo "HEAD=$(git rev-parse HEAD 2>&1)"
echo '--- commits ---'
git show -s --format='%H %s' c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89 2>&1 || true
git show -s --format='%H %s' fda69bd4bce4dace3a43566ef6b25e51c8c13e65 2>&1 || true
echo '--- names ---'
git diff --name-status fda69bd4bce4dace3a43566ef6b25e51c8c13e65 c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89 2>&1 || true
echo '--- diff ---'
git diff --no-ext-diff --unified=60 fda69bd4bce4dace3a43566ef6b25e51c8c13e65 c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89 2>&1 || trueLength of output: 1126 🏁 Script executed: #!/bin/bash
set -euo pipefail
head='c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89'
echo '=== parents ==='
git show -s --format='%H%nParents: %P%nSubject: %s' "$head"
echo '=== remote refs related to stack ==='
git for-each-ref --format='%(refname:short) %(objectname) %(subject)' refs/remotes refs/heads |
rg -i 'aic-1952|pr170|host-resource|170' || true
echo '=== candidate ancestry ==='
for rev in "$head^1" "$head^2"; do
printf '%s ' "$rev"
git rev-parse "$rev"
git show -s --format='%H %s' "$rev"
done
echo '=== merge-base ==='
git merge-base "$head^1" "$head^2"
echo '=== parent-to-head changed files ==='
for base in "$head^1" "$head^2"; do
echo "--- $base -> HEAD ---"
git diff --name-status "$base" "$head"
done
echo '=== combined merge diff ==='
git diff-tree --no-commit-id --cc --name-status -r "$head"Length of output: 1724 🏁 Script executed: #!/bin/bash
set -euo pipefail
repo='ai-dynamo/aisimulate'
pr='170'
echo '=== pull request metadata ==='
gh api "repos/$repo/pulls/$pr" \
--jq '{number, state, draft, base: {ref: .base.ref, sha: .base.sha}, head: {ref: .head.ref, sha: .head.sha}, merge_commit_sha}'
echo '=== changed files and patches ==='
gh api --paginate "repos/$repo/pulls/$pr/files?per_page=100" \
--jq '.[] | {filename, status, additions, deletions, patch}'
echo '=== pull request commits ==='
gh api --paginate "repos/$repo/pulls/$pr/commits?per_page=100" \
--jq '.[] | {sha, parents: [.parents[].sha], message: .commit.message}'Length of output: 11873 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== supervision outline ==='
ast-grep outline python/aisimulate/src/aisimulate/supervision.py --items all --type function
echo '=== supervision helper region ==='
nl -ba python/aisimulate/src/aisimulate/supervision.py | sed -n '1,250p'
echo '=== search lifecycle region ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1120,1260p'
echo '=== cleanup helper definitions and call sites ==='
rg -n -C 5 '\b(terminate_pool|close_pool)\s*\(' python/aisimulate/src tests
echo '=== worker and manager private-member access ==='
rg -n -C 4 '(_processes|_executor_manager_thread|manager_thread)' python/aisimulate/src/aisimulateLength of output: 34131 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== resource scheduler lifecycle ==='
nl -ba python/aisimulate/src/aisimulate/resource_scheduler.py | sed -n '1,160p'
echo '=== supervision tests around pool cleanup ==='
nl -ba tests/test_resource_supervision.py | sed -n '280,390p'
echo '=== sweeper timeout and lifecycle paths ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1230,1425p'
echo '=== lifecycle test support and changed tests ==='
nl -ba tests/sweeper/test_search.py | sed -n '250,590p'
echo '=== imports for changed modules ==='
sed -n '1,80p' python/aisimulate/src/aisimulate/resource_scheduler.py
sed -n '1,90p' python/aisimulate/src/aisimulate/sweeper/search.pyLength of output: 43577
You are interacting with an AI system. |
Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>
c56b38e to
ef372a9
Compare
|
@coderabbitai full review Please review current head Fresh Codex self-review of this exact head found no remaining actionable issue in timeout replacement, orchestration failure cleanup, graceful shutdown/escalation, shared scheduler callers, or regression coverage. Lint/format, documentation links, ownership coverage, and whitespace checks pass. Native build, local tests, and Fast CI are being refreshed; Full CI will follow the current-head review and Fast CI. |
|
@coderabbitai The automatic review still displays the obsolete Current-head validation is complete locally: 721 passed, 5 expected optional Dynamo packaging skips, using a native extension rebuilt from this head. Fast CI also passed. Fresh Codex self-review found no remaining actionable issue. GitHub reports the PR approved and mergeable with all existing review threads resolved. Full CI will run after the current-head bot review. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='ef372a936c2fcddefdc80be0babcc798d2ad4db5'
base='2832e6b8756b8b83f18aee2f8e8fcaf23eea8675'
echo '=== revision verification ==='
git rev-parse HEAD
git show -s --format='%H%nParents: %P%nSubject: %s' "$head"
git show -s --format='%H%nSubject: %s' "$base"
echo '=== changed files ==='
git diff --name-status "$base" "$head"
echo '=== exact diff ==='
git diff --no-ext-diff --unified=100 "$base" "$head"
echo '=== changed Python outlines ==='
for f in \
python/aisimulate/src/aisimulate/supervision.py \
python/aisimulate/src/aisimulate/sweeper/search.py \
tests/sweeper/test_search.py; do
if test -f "$f"; then
echo "--- $f ---"
ast-grep outline "$f" --items all --type function,class
fi
done
echo '=== cleanup helper definitions and call sites ==='
rg -n -C 8 '\b(terminate_pool|close_pool)\s*\(' \
python/aisimulate/src tests
echo '=== Sweeper process-pool lifecycle references ==='
rg -n -C 6 '\b(ProcessPoolExecutor|_parallel_pool|parallel_pool|shutdown|submit)\b' \
python/aisimulate/src/aisimulate/sweeper/search.py
echo '=== supervision private executor accesses ==='
rg -n -C 6 '(_processes|_executor_manager_thread|shutdown_lock|manager_thread)' \
python/aisimulate/src/aisimulate/supervision.pyLength of output: 70030 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== Sweeper batch completion and replacement paths ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1280,1465p'
echo '=== shared scheduler ownership and cleanup paths ==='
nl -ba python/aisimulate/src/aisimulate/resource_scheduler.py | sed -n '1,155p'
echo '=== supervision imports and executor context ==='
nl -ba python/aisimulate/src/aisimulate/supervision.py | sed -n '1,165p'
echo '=== fake runner/factory pickling and focused test support ==='
nl -ba tests/sweeper/test_search.py | sed -n '1,165p'
nl -ba tests/sweeper/test_search.py | sed -n '430,575p'
echo '=== executor shutdown API use outside the changed helper ==='
rg -n -C 4 '\.shutdown\s*\(' python/aisimulate/src tests -g '*.py'
echo '=== cleanup helper test coverage ==='
nl -ba tests/test_resource_supervision.py | sed -n '300,375p'
rg -n -C 5 '\b(close_pool|terminate_pool|executor cleanup did not complete|worker survived termination)\b' tests -g '*.py'Length of output: 45514 🏁 Script executed: #!/bin/bash
set -euo pipefail
previous='c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89'
current='ef372a936c2fcddefdc80be0babcc798d2ad4db5'
files=(
docs/local-resources.md
python/aisimulate/src/aisimulate/supervision.py
python/aisimulate/src/aisimulate/sweeper/search.py
tests/sweeper/test_search.py
)
echo '=== prior/current object availability ==='
git cat-file -t "$previous"
git cat-file -t "$current"
echo '=== file identity comparison ==='
for file in "${files[@]}"; do
printf '%s\t' "$file"
git rev-parse "$previous:$file"
git rev-parse "$current:$file"
done
echo '=== content diff for reviewed files ==='
git diff --no-ext-diff --exit-code "$previous" "$current" -- "${files[@]}"
echo '=== differences outside reviewed files ==='
git diff --name-status "$previous" "$current" -- . ':(exclude)docs/local-resources.md' \
':(exclude)python/aisimulate/src/aisimulate/supervision.py' \
':(exclude)python/aisimulate/src/aisimulate/sweeper/search.py' \
':(exclude)tests/sweeper/test_search.py' || trueLength of output: 1453
I reviewed the four-file diff, direct Sweeper timeout and failure cleanup, normal shutdown escalation, shared scheduler callers, and the added real-worker regression coverage. The local checkout did not contain prior commit You are interacting with an AI system. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
head='ef372a936c2fcddefdc80be0babcc798d2ad4db5'
base='2832e6b8756b8b83f18aee2f8e8fcaf23eea8675'
echo '=== revisions and diff ==='
echo "checkout=$(git rev-parse HEAD)"
git show -s --format='HEAD %H%n%P%n%s' "$head" || true
git show -s --format='BASE %H%n%s' "$base" || true
git diff --name-status "$base" "$head" || true
git diff --no-ext-diff --unified=70 "$base" "$head" || true
echo '=== PR API files fallback ==='
gh api --paginate 'repos/ai-dynamo/aisimulate/pulls/170/files?per_page=100' \
--jq '.[] | {filename, status, additions, deletions, patch}'
echo '=== outlines ==='
ast-grep outline python/aisimulate/src/aisimulate/supervision.py --items all --type function
ast-grep outline python/aisimulate/src/aisimulate/sweeper/search.py --items all --type function,class
ast-grep outline tests/sweeper/test_search.py --items all --type function,class
echo '=== helper definitions, callers, and Sweeper lifecycle ==='
rg -n -C 10 '\b(terminate_pool|close_pool)\s*\(' python/aisimulate/src tests
rg -n -C 10 '\b(ProcessPoolExecutor|_parallel_pool|parallel_pool|shutdown|submit)\b' \
python/aisimulate/src/aisimulate/sweeper/search.py
rg -n -C 10 '(_processes|_executor_manager_thread|shutdown_lock|manager_thread)' \
python/aisimulate/src/aisimulate/supervision.py
echo '=== changed test areas ==='
rg -n -C 10 '(ignore|SIGTERM|finalizer|terminate_pool|close_pool|reap|replacement|orchestration)' \
tests/sweeper/test_search.pyLength of output: 80506 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== direct Sweeper batch loop ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1288,1455p'
echo '=== direct Sweeper run lifecycle context ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1455,1605p'
echo '=== shared resource scheduler lifecycle ==='
nl -ba python/aisimulate/src/aisimulate/resource_scheduler.py | sed -n '1,155p'
echo '=== process cleanup helper tests ==='
nl -ba tests/test_resource_supervision.py | sed -n '300,375p'
echo '=== executor implementation version ==='
python - <<'PY'
import concurrent.futures.process
import inspect
import sys
print(sys.version)
print(inspect.getsourcefile(concurrent.futures.process))
PY
echo '=== all direct Sweeper invocation paths ==='
rg -n -C 5 '_replace_pool\s*\(|_pool_lifecycle\s*\(|_terminate_pool\s*\(' \
python/aisimulate/src/aisimulate/sweeper/search.pyLength of output: 32017
I verified the exact four-file diff. I reviewed direct Sweeper timeout replacement, orchestration-failure cleanup, successful-sweep cleanup, executor-manager shutdown ordering, shared scheduler callers, and the spawned-worker regressions. You are interacting with an AI system. |
|
Merge closeout for
#170 is not merged. Preserve its narrow scope by landing #259 first, refreshing this branch onto main, then passing current-head review and CI before a SHA-guarded squash merge. The failed gate is not waived. |
…i-dynamo#170) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>
Direct
Sweepercallers can start a replacement pool while timed-out workers are still alive, or hang while shutting down a completed sweep. This change uses the bounded worker cleanup introduced by #169 for those callers and fixes a shutdown-lock deadlock in the shared helper, so stuck finalizers cannot prevent escalation to kill.#169 has merged. This PR now targets
mainand contains only four files:terminate_poolbefore replacement and after orchestration failures, andclose_poolafter successful direct sweeps. Failed cleanup prevents replacement.Validation
Head:
ef372a936c2fcddefdc80be0babcc798d2ad4db5, based onmainat2832e6b8756b8b83f18aee2f8e8fcaf23eea8675.All four changed files are byte-for-byte identical to the previously approved head
c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89. The rebase removes duplicated parent history after #169's squash merge.test_nightly_versions_are_unique_date_ordered_and_stable_across_retries, becauseshutil.which("node")returnsNonein the Python-only containers. Each shard otherwise passed 1,974 tests plus 132 subtests. The aggregate gate failed as required.This does not qualify the original Mac workload; that remains #174.
Tracks AIC-1953.