Skip to content

fix(sweeper): bound worker cleanup before pool replacement - #170

Merged
jasonqinzhou merged 1 commit into
mainfrom
codex/aic-1953-supervised-execution
Sep 18, 2026
Merged

jasonqinzhou merged 1 commit into
mainfrom
codex/aic-1953-supervised-execution

Conversation

@jasonqinzhou

@jasonqinzhou jasonqinzhou commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Direct Sweeper callers can start a replacement pool while timed-out workers are still alive, or hang while shutting down a completed sweep. This change uses the bounded worker cleanup introduced by #169 for those callers and fixes a shutdown-lock deadlock in the shared helper, so stuck finalizers cannot prevent escalation to kill.

#169 has merged. This PR now targets main and contains only four files:

  • Use terminate_pool before replacement and after orchestration failures, and close_pool after successful direct sweeps. Failed cleanup prevents replacement.
  • Avoid calling executor shutdown again while its manager may hold the shutdown lock joining a hung finalizer. The manager remains the sole worker-join owner; survivors are killed before final executor shutdown.
  • Exercise real spawned workers that ignore SIGTERM, normal finalizers, and hung finalizers. Preserve completed candidate results and verify that workers have been reaped.
  • Document the cleanup guarantee for direct Sweeper callers, including custom factories without resource admission. Sequential Runner execution remains outside this process-pool guarantee.

Validation

Head: ef372a936c2fcddefdc80be0babcc798d2ad4db5, based on main at 2832e6b8756b8b83f18aee2f8e8fcaf23eea8675.

All four changed files are byte-for-byte identical to the previously approved head c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89. The rebase removes duplicated parent history after #169's squash merge.

This does not qualify the original Mac workload; that remains #174.

Tracks AIC-1953.

@copy-pr-bot

copy-pr-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@linear-code

linear-code Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

@coderabbitai

coderabbitai Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 384934e4-5edb-4bf9-ba34-df5595864484

📥 Commits

Reviewing files that changed from the base of the PR and between 2832e6b and ef372a9.

📒 Files selected for processing (4)
  • docs/local-resources.md
  • python/aisimulate/src/aisimulate/supervision.py
  • python/aisimulate/src/aisimulate/sweeper/search.py
  • tests/sweeper/test_search.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • ai-dynamo/dynamo (manual)
  • ai-dynamo/aiconfigurator (manual)

Included review availability: Your plan provides up to 12 included reviews per hour; 2 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (20)
  • GitHub Check: Prediction Regression / Collect prediction snapshot (old)
  • GitHub Check: Prediction Regression / Collect prediction snapshot (new)
  • GitHub Check: Platform Wheels / Build wheels (macosx_arm64)
  • GitHub Check: Platform Wheels / Build wheels (manylinux_2_28_x86_64)
  • GitHub Check: Platform Wheels / Build wheels (manylinux_2_28_aarch64)
  • GitHub Check: Collector Data / Perf data sanity (informational)
  • GitHub Check: Collector Data / Check collector data
  • GitHub Check: Rust (amd64)
  • GitHub Check: Application Test Wheel (arm64)
  • GitHub Check: Python Dependency Licenses
  • GitHub Check: Release Artifact Contract (arm64)
  • GitHub Check: Application Test Wheel (amd64)
  • GitHub Check: Release Artifact Contract (amd64)
  • GitHub Check: Rust (arm64)
  • GitHub Check: Engine Golden Regression
  • GitHub Check: Python 3.13 compatibility
  • GitHub Check: Rust feature modes
  • GitHub Check: Python 3.11 compatibility
  • GitHub Check: Public API Rust (arm64)
  • GitHub Check: Public API Rust (amd64)
🧰 Additional context used
📓 Path-based instructions (5)
Check unified CLI, Replay, Sweeper, and orchestration behavior together.

⚙️ CodeRabbit configuration file

Files:

  • python/aisimulate/src/aisimulate/supervision.py
  • python/aisimulate/src/aisimulate/sweeper/search.py
Require coverage of the changed behavior and its negative or boundary cases.

⚙️ CodeRabbit configuration file

Files:

  • tests/sweeper/test_search.py
Check commands, defaults, supported runtimes, public names, and claims against executable behavior.

⚙️ CodeRabbit configuration file

Files:

  • docs/local-resources.md
Read REVIEW.md before commenting.

⚙️ CodeRabbit configuration file

Files:

  • python/aisimulate/src/aisimulate/supervision.py
  • python/aisimulate/src/aisimulate/sweeper/search.py
  • docs/local-resources.md
  • tests/sweeper/test_search.py
Before making any change under: `python/aisimulate/src/aiconfigurator/generator/**` MUST read: `python/aisimulate/.claude/rules/generator-development.md` Before making any change under `python/aisimulate/collector/**` MUST read: `python/ais...

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • python/aisimulate/src/aisimulate/supervision.py
  • python/aisimulate/src/aisimulate/sweeper/search.py
  • docs/local-resources.md
  • tests/sweeper/test_search.py
🪛 LanguageTool
docs/local-resources.md

[style] ~146-~146: Consider using “who” when you are referring to people instead of objects.
Context: ...izers a grace period, then stop workers that fail to exit. Cleanup failure raises an...

(THAT_WHO)

🔀 Multi-repo context ai-dynamo/dynamo, ai-dynamo/aiconfigurator

Linked repositories findings

ai-dynamo/dynamo

  • Dynamo pins aisimulate==0.12.0 in both project dependencies and container requirements (pyproject.toml:17, container/deps/requirements.aisimulate.txt:5). Existing installations will not receive this cleanup fix until a compatible AISimulate release and/or pin update. [::ai-dynamo/dynamo::]
  • The public integration invokes Sweeper.run with DynamoReplayRunnerFactory; the factory consumes serializable ReplaySpec values and does not import the changed supervision helpers (pyproject.toml:121, components/src/dynamo/replay/simulation.py:55-95). No API mismatch was found. [::ai-dynamo/dynamo::]
  • Downstream documentation confirms that each Sweeper.run owns fresh worker pools and that parallel evaluations use spawned processes (docs/fern/pages/developer-guide/knowledge-base/modular-components/ai-simulate-experimental/sweeper-experimental/overview.md:34-52, quickstart.md:18-28). The new pool cleanup semantics therefore directly affect Dynamo’s documented integration. [::ai-dynamo/dynamo::]

ai-dynamo/aiconfigurator

  • AIC’s migration guide directs users from deprecated sweep_agg/sweep_disagg/sweep_afd APIs to aisimulate.sweeper.Sweeper, using either EngineReplayRunnerFactory or a Dynamo runner factory (docs/aisimulate_migration.md:48-67). No direct references to the changed cleanup helpers were found. [::ai-dynamo/aiconfigurator::]
  • The legacy sweep entry points are explicitly compatibility wrappers emitting migration warnings (src/aiconfigurator/sdk/sweep.py:451, 1284, 1595; src/aiconfigurator/deprecation.py:42-60). This change does not require an AIC API update, but the exact aisimulate==0.12.0 migration baseline should remain aligned with the released cleanup behavior. [::ai-dynamo/aiconfigurator::]
🔇 Additional comments (4)
python/aisimulate/src/aisimulate/supervision.py (1)

124-125: LGTM!

python/aisimulate/src/aisimulate/sweeper/search.py (1)

1199-1201: LGTM!

Also applies to: 1220-1225

tests/sweeper/test_search.py (1)

352-353: LGTM!

Also applies to: 399-400, 440-574

docs/local-resources.md (1)

143-150: LGTM!


📝 Summary

Risk: Medium

Human attention should focus on:

  1. terminate_pool escalation and manager-thread ownership.
  2. Pool replacement after timeouts and orchestration failures.
  3. The contract for custom factories and sequential execution.

Changed behavior

  • Sweeper calls terminate_pool before replacing a timed-out pool.
  • Sweeper calls terminate_pool when orchestration fails.
  • Successful sweeps call close_pool.
  • terminate_pool terminates descendants, signals workers, waits for the executor manager, kills surviving workers, and refuses replacement if cleanup remains incomplete.
  • close_pool allows a bounded finalizer grace period, then escalates through terminate_pool.
  • Completed results remain available when shutdown is bounded.
  • Direct Sweeper callers receive this cleanup behavior even without resource admission.
  • Custom factories do not receive memory admission or RSS monitoring.
  • Sequential Runner execution remains outside the process-pool cleanup guarantee.

Areas for review

  • terminate_pool uses private ProcessPoolExecutor state, including _processes and _executor_manager_thread.
  • Cleanup failure raises instead of creating a replacement pool. Callers must handle this failure path.
  • The implementation uses fixed supervision grace periods. The documented resource shutdown setting does not appear to control this helper.
  • Descendant discovery occurs before worker termination. Descendants created later may not be included in that termination pass.

Evidence

  • The supplied validation reports 721 tests passed, five optional tests skipped, and Fast CI passed.
  • Spawned-worker regressions cover SIGTERM-resistant workers, hung finalizers, completed-result preservation, replacement overlap, and worker reaping.
  • CodeRabbit and Codex reported no actionable findings. This is not independent human approval.
  • No current review severity counts or independent human review findings were supplied.
  • The original Mac workload remains unqualified and is tracked separately.

Walkthrough

Pool cleanup now uses shared supervision helpers. Timed-out pools are reaped before replacement, normal cleanup allows bounded finalization, and regression tests cover worker termination, result preservation, and hung finalizers.

Changes

Bounded process-pool cleanup

Layer / File(s) Summary
Termination helper ordering
python/aisimulate/src/aisimulate/supervision.py
terminate_pool waits for the executor manager before final shutdown.
Sweeper pool lifecycle integration
python/aisimulate/src/aisimulate/sweeper/search.py
Sweeper delegates timed-out pool termination to terminate_pool and normal cleanup to close_pool.
Lifecycle regression coverage and documentation
tests/sweeper/test_search.py, docs/local-resources.md
Tests cover non-blocking normal shutdown, awaited replacement pools, worker reaping, completed-result preservation, and bounded hung finalizers. Documentation describes the direct Sweeper cleanup behavior and its limits.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Unblocks: 1 PR

Merge Risk: ⚪ Minimal · up to ef372

The bounded Sweeper cleanup change has no identified current behavior that would block merging.

🚥 Pre-merge checks | ✅ 8
✅ Passed checks (8 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cross-Layer Contract ✅ Passed PASS: The PR changes worker lifecycle behavior only. The authoritative diff has four files, with no Rust, CLI schema, configuration, result-schema, serialization, or public declaration changes. `Sweep…
Modeling And Data Evidence ✅ Passed PASS: The authoritative diff changes process-pool cleanup, worker lifecycle handling, documentation, and lifecycle regression tests. It does not introduce formulas, selection/model logic, performance …
Compatibility Boundaries ✅ Passed The PR changes only Sweeper pool cleanup, supervision shutdown ordering, tests, and local-resource documentation. No Rust, packaging, schema, binding, version, or public export files changed. Public d…
Review Evidence ✅ Passed The description names the pytest command and reports 65 passed, 721 passed, 5 skipped, Fast CI success, and Full CI status. It includes boundary evidence: the replacement regression fails against …
Title check ✅ Passed The title precisely states the behavioral change: worker cleanup is bounded before pool replacement.
Description check ✅ Passed The description clearly explains the problem, implementation, affected consumers, regression coverage, validation results, CI status, scope, and tracking issue. It does not use every template heading,…

Comment @coderabbitai help to get the list of available commands.

Comment thread python/aisimulate/pyproject.toml Resolved
@jasonqinzhou

Copy link
Copy Markdown
Contributor Author

The requested overlap audit compared #170 at 156ab0712fca25876a012e6e0e25f91f91eac107 with #169 at 9d69d781b912382275a30d40c7da7d00a63c97c5.

#169 now includes the CLI/SDK supervisor, owned-process memory monitoring, runtime thread limits, deadlines, checkpoint evidence, launchers, and the supervision regression tests. Carrying the old #170 implementation forward would duplicate those changes and conflict with the newer candidate recovery and output contracts.

The remaining concrete gap is direct low-level Sweeper process-pool cleanup: the non-resource-aware path terminates workers without waiting before replacement and waits without a bound during normal shutdown. I will rebase this draft onto #169 and retain only reuse of #169's existing terminate_pool/close_pool helpers, focused lifecycle regressions, and the corresponding documentation. The older optimizer.parallelism: auto schema change and fixed configuration-size cutoff are excluded; #169 already adjusts execution concurrency automatically.

The PR will remain a draft. #174's integrated workload qualification remains separate.

@jasonqinzhou

Copy link
Copy Markdown
Contributor Author

The new real-worker shutdown regression exposed a concrete gap inherited from #169's cleanup helper at 9d69d781:

[P1] Do not call executor shutdown again before escalating stuck workers. close_pool starts graceful shutdown, and Python's executor manager holds its shutdown lock while joining workers. If a Runner finalizer hangs and ignores SIGTERM, terminate_pool calls pool.shutdown(wait=False) again before sending SIGKILL. That call waits for the same lock, so cleanup cannot reach its kill fallback. The regression's completed results remain blocked until the controlled finalizer returns.

Keep the existing executor manager as the sole worker-join owner, signal workers, wait only for the bounded grace period, kill survivors, and finish executor shutdown after the manager exits. This belongs in the narrowed cleanup follow-up and will be covered together with the direct Sweeper lifecycle tests.

@jasonqinzhou
jasonqinzhou force-pushed the codex/aic-1953-supervised-execution branch from 156ab07 to 0a67422 Compare September 18, 2026 00:02
@jasonqinzhou jasonqinzhou changed the title fix(resources): supervise local predict and recommend execution fix(sweeper): bound worker cleanup before pool replacement Sep 18, 2026
@jasonqinzhou

jasonqinzhou commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Updated in 0a67422b109d5b0264bc955a5f6ad35a9148a122, rebased onto #169 at f7a4210c30948b79ba24925551c452fc58b129a2.

The PR now has one commit and four changed files. It retains only bounded direct Sweeper pool cleanup, the shared helper's shutdown-lock correction, real-worker regressions, and the documentation for that behavior. The duplicated CLI/SDK supervisor, resource policies, launchers, and result changes are supplied by #169.

The cleanup finding above is fixed: shutdown is completed after the manager exits, so a hung finalizer cannot block SIGKILL escalation. The complete Sweeper/resource/scheduler/supervision/unified CLI run passed 721 tests, with 5 expected optional Dynamo integration skips. The replacement regression also fails against the original #169 implementation and passes with the fix. Configured lint/format, docs links, ownership coverage, and whitespace checks pass. Final Codex self-review found no remaining actionable issue in this exact four-file diff.

The draft remains based on #169. Current-head Fast CI passed; Full CI and human review have not been completed for this head. #174 needs its qualification branch refreshed separately against the narrowed stack.

@jasonqinzhou
jasonqinzhou marked this pull request as ready for review September 18, 2026 00:05
@jasonqinzhou
jasonqinzhou requested review from a team as code owners September 18, 2026 00:05
@jasonqinzhou

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

Please review exact head c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89, stacked on #169 at fda69bd4bce4dace3a43566ef6b25e51c8c13e65. Automatic review skipped the non-default target branch, so this is an explicit review request.

The complete PR diff is four files: direct Sweeper pool cleanup now uses the existing bounded cleanup helpers; the shared termination helper avoids a shutdown-lock deadlock before killing stuck workers; real spawned-worker regressions cover termination resistance, normal finalizers, hung finalizers, and preserved completed results; documentation describes the exact scope.

Codex re-read the full current diff and cleanup/error paths and found no remaining actionable issue. The unchanged application/test code passed 721 local tests with five expected optional Dynamo integration skips before the documentation-only parent update. Focused current-head regression validation is running. Please identify any remaining actionable findings against this exact head; current-head Fast CI, this review, and the Codex review must complete before Full CI dispatch. Human/CODEOWNER approval remains separate.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '=== checked-out revision ==='
git rev-parse HEAD
git status --short
printf '%s\n' '=== exact stacked diff file list ==='
git diff --name-status fda69bd4bce4dace3a43566ef6b25e51c8c13e65 c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89
printf '%s\n' '=== exact stacked diff ==='
git diff --find-renames --find-copies --no-ext-diff --unified=80 \
  fda69bd4bce4dace3a43566ef6b25e51c8c13e65 \
  c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89

Length of output: 980


🏁 Script executed:

#!/bin/bash
set -u
echo "HEAD=$(git rev-parse HEAD 2>&1)"
echo '--- commits ---'
git show -s --format='%H %s' c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89 2>&1 || true
git show -s --format='%H %s' fda69bd4bce4dace3a43566ef6b25e51c8c13e65 2>&1 || true
echo '--- names ---'
git diff --name-status fda69bd4bce4dace3a43566ef6b25e51c8c13e65 c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89 2>&1 || true
echo '--- diff ---'
git diff --no-ext-diff --unified=60 fda69bd4bce4dace3a43566ef6b25e51c8c13e65 c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89 2>&1 || true

Length of output: 1126


🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89'
echo '=== parents ==='
git show -s --format='%H%nParents: %P%nSubject: %s' "$head"
echo '=== remote refs related to stack ==='
git for-each-ref --format='%(refname:short) %(objectname) %(subject)' refs/remotes refs/heads |
  rg -i 'aic-1952|pr170|host-resource|170' || true
echo '=== candidate ancestry ==='
for rev in "$head^1" "$head^2"; do
  printf '%s ' "$rev"
  git rev-parse "$rev"
  git show -s --format='%H %s' "$rev"
done
echo '=== merge-base ==='
git merge-base "$head^1" "$head^2"
echo '=== parent-to-head changed files ==='
for base in "$head^1" "$head^2"; do
  echo "--- $base -> HEAD ---"
  git diff --name-status "$base" "$head"
done
echo '=== combined merge diff ==='
git diff-tree --no-commit-id --cc --name-status -r "$head"

Length of output: 1724


🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo='ai-dynamo/aisimulate'
pr='170'
echo '=== pull request metadata ==='
gh api "repos/$repo/pulls/$pr" \
  --jq '{number, state, draft, base: {ref: .base.ref, sha: .base.sha}, head: {ref: .head.ref, sha: .head.sha}, merge_commit_sha}'
echo '=== changed files and patches ==='
gh api --paginate "repos/$repo/pulls/$pr/files?per_page=100" \
  --jq '.[] | {filename, status, additions, deletions, patch}'
echo '=== pull request commits ==='
gh api --paginate "repos/$repo/pulls/$pr/commits?per_page=100" \
  --jq '.[] | {sha, parents: [.parents[].sha], message: .commit.message}'

Length of output: 11873


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== supervision outline ==='
ast-grep outline python/aisimulate/src/aisimulate/supervision.py --items all --type function
echo '=== supervision helper region ==='
nl -ba python/aisimulate/src/aisimulate/supervision.py | sed -n '1,250p'
echo '=== search lifecycle region ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1120,1260p'
echo '=== cleanup helper definitions and call sites ==='
rg -n -C 5 '\b(terminate_pool|close_pool)\s*\(' python/aisimulate/src tests
echo '=== worker and manager private-member access ==='
rg -n -C 4 '(_processes|_executor_manager_thread|manager_thread)' python/aisimulate/src/aisimulate

Length of output: 34131


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== resource scheduler lifecycle ==='
nl -ba python/aisimulate/src/aisimulate/resource_scheduler.py | sed -n '1,160p'
echo '=== supervision tests around pool cleanup ==='
nl -ba tests/test_resource_supervision.py | sed -n '280,390p'
echo '=== sweeper timeout and lifecycle paths ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1230,1425p'
echo '=== lifecycle test support and changed tests ==='
nl -ba tests/sweeper/test_search.py | sed -n '250,590p'
echo '=== imports for changed modules ==='
sed -n '1,80p' python/aisimulate/src/aisimulate/resource_scheduler.py
sed -n '1,90p' python/aisimulate/src/aisimulate/sweeper/search.py

Length of output: 43577


@jasonqinzhou: No actionable findings in #170 at c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89.

You are interacting with an AI system.

Base automatically changed from codex/aic-1952-host-resource-budgets to main September 18, 2026 00:21
@jasonqinzhou
jasonqinzhou requested review from a team as code owners September 18, 2026 00:21
Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>
@jasonqinzhou
jasonqinzhou force-pushed the codex/aic-1953-supervised-execution branch from c56b38e to ef372a9 Compare September 18, 2026 00:27
@jasonqinzhou

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

Please review current head ef372a936c2fcddefdc80be0babcc798d2ad4db5 against main (2832e6b8756b8b83f18aee2f8e8fcaf23eea8675). #169 has merged, and this PR has been rebased to remove duplicated parent history. The diff is again exactly four files (+158/-16); every changed file is byte-for-byte identical to the approved and previously reviewed head c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89.

Fresh Codex self-review of this exact head found no remaining actionable issue in timeout replacement, orchestration failure cleanup, graceful shutdown/escalation, shared scheduler callers, or regression coverage. Lint/format, documentation links, ownership coverage, and whitespace checks pass. Native build, local tests, and Fast CI are being refreshed; Full CI will follow the current-head review and Fast CI.

@jasonqinzhou

Copy link
Copy Markdown
Contributor Author

@coderabbitai The automatic review still displays the obsolete c56b38e5 comparison triggered by #169's merge. Please inspect the current four-file diff at ef372a936c2fcddefdc80be0babcc798d2ad4db5 against 2832e6b8756b8b83f18aee2f8e8fcaf23eea8675 and report any actionable findings for that exact head. The changed files are identical to your previously reviewed version; the stale ancestry is removed. If local history is shallow, gh api repos/ai-dynamo/aisimulate/pulls/170/files exposes the current diff.

Current-head validation is complete locally: 721 passed, 5 expected optional Dynamo packaging skips, using a native extension rebuilt from this head. Fast CI also passed. Fresh Codex self-review found no remaining actionable issue. GitHub reports the PR approved and mergeable with all existing review threads resolved. Full CI will run after the current-head bot review.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='ef372a936c2fcddefdc80be0babcc798d2ad4db5'
base='2832e6b8756b8b83f18aee2f8e8fcaf23eea8675'

echo '=== revision verification ==='
git rev-parse HEAD
git show -s --format='%H%nParents: %P%nSubject: %s' "$head"
git show -s --format='%H%nSubject: %s' "$base"
echo '=== changed files ==='
git diff --name-status "$base" "$head"
echo '=== exact diff ==='
git diff --no-ext-diff --unified=100 "$base" "$head"

echo '=== changed Python outlines ==='
for f in \
  python/aisimulate/src/aisimulate/supervision.py \
  python/aisimulate/src/aisimulate/sweeper/search.py \
  tests/sweeper/test_search.py; do
  if test -f "$f"; then
    echo "--- $f ---"
    ast-grep outline "$f" --items all --type function,class
  fi
done

echo '=== cleanup helper definitions and call sites ==='
rg -n -C 8 '\b(terminate_pool|close_pool)\s*\(' \
  python/aisimulate/src tests

echo '=== Sweeper process-pool lifecycle references ==='
rg -n -C 6 '\b(ProcessPoolExecutor|_parallel_pool|parallel_pool|shutdown|submit)\b' \
  python/aisimulate/src/aisimulate/sweeper/search.py

echo '=== supervision private executor accesses ==='
rg -n -C 6 '(_processes|_executor_manager_thread|shutdown_lock|manager_thread)' \
  python/aisimulate/src/aisimulate/supervision.py

Length of output: 70030


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== Sweeper batch completion and replacement paths ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1280,1465p'

echo '=== shared scheduler ownership and cleanup paths ==='
nl -ba python/aisimulate/src/aisimulate/resource_scheduler.py | sed -n '1,155p'

echo '=== supervision imports and executor context ==='
nl -ba python/aisimulate/src/aisimulate/supervision.py | sed -n '1,165p'

echo '=== fake runner/factory pickling and focused test support ==='
nl -ba tests/sweeper/test_search.py | sed -n '1,165p'
nl -ba tests/sweeper/test_search.py | sed -n '430,575p'

echo '=== executor shutdown API use outside the changed helper ==='
rg -n -C 4 '\.shutdown\s*\(' python/aisimulate/src tests -g '*.py'

echo '=== cleanup helper test coverage ==='
nl -ba tests/test_resource_supervision.py | sed -n '300,375p'
rg -n -C 5 '\b(close_pool|terminate_pool|executor cleanup did not complete|worker survived termination)\b' tests -g '*.py'

Length of output: 45514


🏁 Script executed:

#!/bin/bash
set -euo pipefail

previous='c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89'
current='ef372a936c2fcddefdc80be0babcc798d2ad4db5'
files=(
  docs/local-resources.md
  python/aisimulate/src/aisimulate/supervision.py
  python/aisimulate/src/aisimulate/sweeper/search.py
  tests/sweeper/test_search.py
)

echo '=== prior/current object availability ==='
git cat-file -t "$previous"
git cat-file -t "$current"

echo '=== file identity comparison ==='
for file in "${files[@]}"; do
  printf '%s\t' "$file"
  git rev-parse "$previous:$file"
  git rev-parse "$current:$file"
done

echo '=== content diff for reviewed files ==='
git diff --no-ext-diff --exit-code "$previous" "$current" -- "${files[@]}"

echo '=== differences outside reviewed files ==='
git diff --name-status "$previous" "$current" -- . ':(exclude)docs/local-resources.md' \
  ':(exclude)python/aisimulate/src/aisimulate/supervision.py' \
  ':(exclude)python/aisimulate/src/aisimulate/sweeper/search.py' \
  ':(exclude)tests/sweeper/test_search.py' || true

Length of output: 1453


@jasonqinzhou No actionable findings in #170 at ef372a936c2fcddefdc80be0babcc798d2ad4db5 against main at 2832e6b8756b8b83f18aee2f8e8fcaf23eea8675.

I reviewed the four-file diff, direct Sweeper timeout and failure cleanup, normal shutdown escalation, shared scheduler callers, and the added real-worker regression coverage.

The local checkout did not contain prior commit c56b38e5ca192aa0b4dafc4f0a77c0c8fe52ad89, so I could not independently verify the claimed byte identity.

You are interacting with an AI system.

@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='ef372a936c2fcddefdc80be0babcc798d2ad4db5'
base='2832e6b8756b8b83f18aee2f8e8fcaf23eea8675'

echo '=== revisions and diff ==='
echo "checkout=$(git rev-parse HEAD)"
git show -s --format='HEAD %H%n%P%n%s' "$head" || true
git show -s --format='BASE %H%n%s' "$base" || true
git diff --name-status "$base" "$head" || true
git diff --no-ext-diff --unified=70 "$base" "$head" || true

echo '=== PR API files fallback ==='
gh api --paginate 'repos/ai-dynamo/aisimulate/pulls/170/files?per_page=100' \
  --jq '.[] | {filename, status, additions, deletions, patch}'

echo '=== outlines ==='
ast-grep outline python/aisimulate/src/aisimulate/supervision.py --items all --type function
ast-grep outline python/aisimulate/src/aisimulate/sweeper/search.py --items all --type function,class
ast-grep outline tests/sweeper/test_search.py --items all --type function,class

echo '=== helper definitions, callers, and Sweeper lifecycle ==='
rg -n -C 10 '\b(terminate_pool|close_pool)\s*\(' python/aisimulate/src tests
rg -n -C 10 '\b(ProcessPoolExecutor|_parallel_pool|parallel_pool|shutdown|submit)\b' \
  python/aisimulate/src/aisimulate/sweeper/search.py
rg -n -C 10 '(_processes|_executor_manager_thread|shutdown_lock|manager_thread)' \
  python/aisimulate/src/aisimulate/supervision.py

echo '=== changed test areas ==='
rg -n -C 10 '(ignore|SIGTERM|finalizer|terminate_pool|close_pool|reap|replacement|orchestration)' \
  tests/sweeper/test_search.py

Length of output: 80506


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== direct Sweeper batch loop ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1288,1455p'

echo '=== direct Sweeper run lifecycle context ==='
nl -ba python/aisimulate/src/aisimulate/sweeper/search.py | sed -n '1455,1605p'

echo '=== shared resource scheduler lifecycle ==='
nl -ba python/aisimulate/src/aisimulate/resource_scheduler.py | sed -n '1,155p'

echo '=== process cleanup helper tests ==='
nl -ba tests/test_resource_supervision.py | sed -n '300,375p'

echo '=== executor implementation version ==='
python - <<'PY'
import concurrent.futures.process
import inspect
import sys
print(sys.version)
print(inspect.getsourcefile(concurrent.futures.process))
PY

echo '=== all direct Sweeper invocation paths ==='
rg -n -C 5 '_replace_pool\s*\(|_pool_lifecycle\s*\(|_terminate_pool\s*\(' \
  python/aisimulate/src/aisimulate/sweeper/search.py

Length of output: 32017


@jasonqinzhou No actionable findings in #170 at ef372a936c2fcddefdc80be0babcc798d2ad4db5 against main at 2832e6b8756b8b83f18aee2f8e8fcaf23eea8675.

I verified the exact four-file diff. I reviewed direct Sweeper timeout replacement, orchestration-failure cleanup, successful-sweep cleanup, executor-manager shutdown ordering, shared scheduler callers, and the spawned-worker regressions.

You are interacting with an AI system.

@jasonqinzhou

Copy link
Copy Markdown
Contributor Author

Merge closeout for ef372a936c2fcddefdc80be0babcc798d2ad4db5:

  • The main-based diff is exactly four files, identical in contents to the approved stacked version. Local validation passed 721 tests with five expected optional Dynamo packaging skips; Fast CI, CodeRabbit, and Codex review pass. GitHub accepts the existing applicable CODEOWNER approval, and all review threads are resolved.
  • Full CI completed with 51 successful jobs, two failed contract shards, and the correctly failed aggregate gate. Both AMD64 and ARM64 failures are the same inherited tests/test_ci_workflow_contracts.py::test_nightly_versions_are_unique_date_ordered_and_stable_across_retries: shutil.which("node") returns None in the Python-only test containers, causing subprocess.run to raise TypeError. Each shard otherwise passed 1,974 tests and 132 subtests, including this PR's regressions.
  • fix(ci): restore nightly validation and historical artifact output #259 already fixes this exact main-branch regression and has passing Full CI. It remains open awaiting applicable CODEOWNER approval, with auto-merge enabled.

#170 is not merged. Preserve its narrow scope by landing #259 first, refreshing this branch onto main, then passing current-head review and CI before a SHA-guarded squash merge. The failed gate is not waived.

@jasonqinzhou
jasonqinzhou merged commit 3ca81b3 into main Sep 18, 2026
63 of 66 checks passed
@jasonqinzhou
jasonqinzhou deleted the codex/aic-1953-supervised-execution branch September 18, 2026 01:11
carlory pushed a commit to carlory/aisimulate that referenced this pull request Oct 30, 2026
…i-dynamo#170)

Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants