feat(aisimulate): move experimental Spica into Dynamo

[tedzhouhk]

Summary

• move Spica source, tests, documentation, configs, images, and tools from ai-dynamo/aiconfigurator@111b093a2a516d6cb2eabac5ad601c95c14ebdbe into Dynamo
• publish AI Simulate as an independent Python distribution with canonical import aisimulate.spica and module CLI python -m aisimulate.spica; do not expose a top-level spica alias, dynamo.profiler.spica alias, console script, or ai-dynamo[spica] extra
• keep package source and tests under aisimulate/, and place runnable configs/tools under the repository-standard examples/aisimulate/spica/ path
• keep Spica explicitly experimental in its package, CLI, examples, release note, and Fern documentation
• build the aisimulate wheel from the same checkout as ai-dynamo, install it explicitly in the existing dynamo-planner image, and retain the required aic-forward-pass,mocker-kvbm-offload runtime features
• route both /aisimulate/ and /examples/aisimulate/ ownership and CI selection to the Dynamo Planner team

Related to #9410.

AIC namespace-doc follow-up: ai-dynamo/aiconfigurator#1411. Earlier migration cleanup landed in ai-dynamo/aiconfigurator#1401 and ai-dynamo/aiconfigurator#1389.

Validation

• PR Planner test selection: 225 passed, 1 skipped across 226 selected tests (pre_merge and planner and gpu_0)
• prior real Rust replay integration validation: 4 passed, including Planner bridge, static replay, KVBM offload, and smart search
• clean wheel namespace smoke confirms aisimulate/spica/** is present and no top-level spica/** alias is published
• installed-package metadata/CLI contract tests: 10 passed
• python -m aisimulate.spica --help shows [EXPERIMENTAL]
• amd64 and arm64 Python 3.12 dependency resolution
• rendered Planner Dockerfile installs the AI Simulate wheel and copies only the test tree; runnable examples arrive through the existing top-level examples copy
• all pre-commit hooks on changed files, including generated CODEOWNERS coverage and pytest marker validation
• fern check: 0 errors; Fern navigation covers the AI Simulate landing page and all nine Spica pages
• source/docs/examples audit finds no old python -m spica, aisimulate/examples/spica, or top-level-package references

Release note

Experimental: Spica is now available from Dynamo's standalone AI Simulate distribution as aisimulate.spica and can be invoked with python -m aisimulate.spica. Its APIs, configuration schema, search results, and deployment outputs may change without a standard deprecation period. It is not production-ready and provides no SLA, accuracy, or configuration-optimality guarantee.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019fa612-16f4-772a-b12f-043d5bb39177.docs.buildwithfern.com/dynamo/dev

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 32.14%
• Overall Coverage: 43.37% (-6.84%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e696c56 | Docs | Datadog PR Page | Give us feedback!}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[coderabbitai]

Walkthrough

Changes

Spica smart search

Layer / File(s)	Summary
Configuration and public API components/src/dynamo/profiler/spica/{__init__.py,__main__.py,_quiet.py,config.py}	Adds the experimental package API, YAML schema, workload and goal validation, import quieting, and module CLI.
Deployment and replay evaluation components/src/dynamo/profiler/spica/{deploy.py,evaluator.py}	Builds aggregated or disaggregated deployment plans and routes trace or synthetic workloads through compatible replay APIs.
KV and parallel configuration modeling components/src/dynamo/profiler/spica/{kv_estimate.py,kv_load.py,model_hw.py,parallel_enum.py,parallel_projection.py}	Adds KV-capacity estimation, load-derived concurrency, hardware resolution, legal parallel enumeration, and configuration projection.
Planner, sampling, and search components/src/dynamo/profiler/spica/{planner.py,load_predictor_sweep.py,sample.py,sampler.py,search.py,search_space.py,score.py}	Adds planner preset expansion, predictor sweeps, sample unrolling, Vizier sampling, cached evaluation, and scalar/Pareto candidate scoring.
Validation and integration coverage components/src/dynamo/profiler/spica/tests/*	Adds unit, integration, replay, packaging, CLI, configuration, enumeration, sampler, scoring, and orchestration tests.
Replay compatibility and packaging components/src/dynamo/replay/*, lib/bindings/python/rust/llm/*, pyproject.toml, container/*, tests/report_pytest_markers.py	Adds AIC estimator and compiled-SDK fallbacks, optional dependencies, runtime wheel features, and import stubs.
Documentation and examples docs/components/profiler/spica/*, examples/profiler/spica/*, docs/index.yml	Documents configuration, traffic, search-space projection, scoring, replay setup, and reproducible sweep examples.

Estimated code review effort: 5 (Critical) | ~120 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly captures the main change: moving experimental Spica into the Dynamo/AI Simulate workflow.
Description check	✅ Passed	The description is detailed and relevant, but it does not follow the required template headings and omits a clear reviewer-start section.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 11

🧹 Nitpick comments (10)

components/src/dynamo/profiler/spica/sampler.py (1)

98-99: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff

In-function imports flagged by path instructions. Path instructions ("Keep all imports at the top of each file; don't add import statements inside functions/classes") require top-level imports; both files instead import heavy optional deps inside function bodies. This appears intentional to keep import dynamo.profiler.spica light (and to let _quiet pin JAX before Vizier/JAX load), but since it hides dependencies please confirm the deferral is deliberate and, if kept, annotate/consolidate the repeated imports.

• components/src/dynamo/profiler/spica/sampler.py#L98-L99: from vizier.service import clients / pyvizier is re-imported in __init__, _update_projection_metadata, observe, observe_infeasible (L224/233/243) — consolidate the repeated pyvizier as vz imports and confirm the lazy vizier import is intended (optional ai-dynamo[spica] dep).
• components/src/dynamo/profiler/spica/load_predictor_sweep.py#L198-L212: from dynamo.planner.core.load.predictors import LOAD_PREDICTORS and from dynamo.planner.offline.trace_data import extract_metrics_from_mooncake are function-local — confirm the lazy dynamo-planner import is intended and document why (Rust runtime + prophet/pmdarima/filterpy).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/profiler/spica/sampler.py` around lines 98 - 99, The
lazy imports are intentional for optional, heavy dependencies, but the repeated
Vizier imports need consolidation. In
components/src/dynamo/profiler/spica/sampler.py:98-99, retain and document the
deferred Vizier import strategy, then reuse a single local/module-level import
across __init__, _update_projection_metadata, observe, and observe_infeasible.
In components/src/dynamo/profiler/spica/load_predictor_sweep.py:198-212, retain
the lazy dynamo-planner imports and document that deferral is required for Rust
runtime and prophet/pmdarima/filterpy dependency loading; no top-level import is
needed there.

Source: Path instructions

lib/bindings/python/rust/llm/aic_callback.rs (1)

32-39: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Redundant unsafe impl Send/Sync — Py<PyAny> is already unconditionally Send + Sync.

PyAicCallback's only field is session: Py<PyAny>. Per pyo3's docs, Py<T> (including Py<PyAny>) implements Send and Sync unconditionally, so the struct would already auto-derive both without any unsafe block. The manual impls here are dead weight that adds unsafe-code audit surface without any functional benefit.

♻️ Proposed removal

-// PyAicCallback enters Python only through Python::with_gil.
-unsafe impl Send for PyAicCallback {}
-unsafe impl Sync for PyAicCallback {}
-
 impl PyAicCallback {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/bindings/python/rust/llm/aic_callback.rs` around lines 32 - 39, Remove
the redundant unsafe Send and Sync implementations for PyAicCallback; its
session: Py<PyAny> field already provides these auto traits. Leave the struct
definition and Python GIL access behavior unchanged.

pyproject.toml (1)

90-102: 🧹 Nitpick | 🔵 Trivial

LGTM! Versions are consistent with container/deps/requirements.planner.txt as the comment requires.

One operational note: tfp-nightly==0.26.0.dev20260626 pins an exact dated nightly build. Nightly/dev channels on PyPI are sometimes pruned after a retention window, which could eventually break fresh installs of ai-dynamo[spica] even though the extra is experimental. Worth keeping an eye on and re-pinning to a stable tensorflow-probability release once Vizier's tfp constraint allows it.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyproject.toml` around lines 90 - 102, The dependency versions are
consistent; no code change is required. Monitor the exact tfp-nightly pin in the
spica dependency set and replace it with a stable tensorflow-probability release
when the google-vizier constraint permits.

lib/bindings/python/tests/replay/test_replay_aic_capacity.py (1)

203-207: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use a raw string for the pytest.raises(match=...) regex.

Ruff (RUF043) flags this: the pattern contains .* but isn't a raw/escaped string, which is ambiguous style even though it currently matches as intended.

🔧 Proposed fix

     with pytest.raises(
         Exception,
-        match="Failed to estimate AIC KV cache capacity.*invalid capacity request",
+        match=r"Failed to estimate AIC KV cache capacity.*invalid capacity request",
     ):

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/bindings/python/tests/replay/test_replay_aic_capacity.py` around lines
203 - 207, Update the match pattern in the pytest.raises assertion surrounding
_run_direct_aic_replay to use a raw string literal, preserving the existing
regex and expected exception behavior.

Source: Linters/SAST tools

components/src/dynamo/profiler/spica/model_hw.py (1)

26-32: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Avoid depending on AIConfigurator private helpers
_estimate_model_weight_bytes and _get_system_config are internal aiconfigurator.generator.naive helpers, so this code is tied to a non-public API and can break on a minor AIConfigurator upgrade without a deprecation path. Add a small compatibility test here, or upstream a public accessor if one exists.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/profiler/spica/model_hw.py` around lines 26 - 32,
Remove the direct dependency on the private helpers _estimate_model_weight_bytes
and _get_system_config in the model hardware profiling flow, replacing them with
a supported public AIConfigurator accessor if available. If no public accessor
exists, add a focused compatibility test covering the required behavior and
isolate the private-helper usage behind that compatibility boundary.

components/src/dynamo/profiler/spica/search.py (1)

145-157: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Log before swallowing broad exceptions.

Both except Exception blocks catch and swallow errors, keeping only type(exc).__name__: exc — no traceback. This is a deliberate design (one bad candidate shouldn't abort the sweep), but it means a genuine bug in unroll_sample/build_deployment/evaluator.evaluate looks identical to an expected candidate failure and is essentially undebuggable after the fact (worker stdout is easy to lose). As per coding guidelines, "Prefer failing fast over hiding errors... always re-raise after logging if catching Exception broadly" — here re-raising isn't appropriate (by design), but logging with logger.exception(...) (lazy formatting) before returning would preserve the traceback for postmortem debugging without changing the resilience behavior.

♻️ Proposed fix

     except InfeasibleKVCapacity as exc:
         return None, None, "infeasible", f"candidate KV capacity infeasible: {exc}"
     except Exception as exc:
+        logger.exception("candidate build failed")
         return (
             None,
             None,
             "failed",
             f"candidate build failed: {type(exc).__name__}: {exc}",
         )
     ...
     try:
         report = evaluator.evaluate(plan, concurrency_override=concurrency)
     except Exception as exc:  # one candidate failing must not abort the sweep
+        logger.exception("replay failed")
         return None, None, "failed", f"replay failed: {type(exc).__name__}: {exc}"

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/profiler/spica/search.py` around lines 145 - 157, Add
logger.exception calls in both broad Exception handlers surrounding candidate
construction and evaluator.evaluate, using lazy formatting and including the
relevant operation context and exception details before returning the existing
failed results. Preserve the current non-aborting sweep behavior and return
values.

Source: Coding guidelines

docs/components/profiler/spica/traffic.md (1)

114-116: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a language to the fenced code block.

markdownlint (MD040) flags this fence as missing a language, unlike the other math blocks in this doc which use text.

📝 Suggested fix

-```
+```text
 num_requests = max(1, round(num_request_ratio * load))

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @docs/components/profiler/spica/traffic.md around lines 114 - 116, Add the
text language identifier to the fenced code block documenting
resolved_request_count, changing the opening fence to use text while preserving
the existing formula and closing fence.

</details>

<!-- cr-comment:v1:a842e6be4aceaa02b1c529ca -->

_Source: Linters/SAST tools_

</blockquote></details>
<details>
<summary>components/src/dynamo/profiler/spica/tests/test_sampler.py (1)</summary><blockquote>

`274-274`: _📐 Maintainability & Code Quality_ | _🔵 Trivial_ | _💤 Low value_

**Minor: prefer `next(iter(...))` over single-element slice.**

Ruff RUF015 flags `list(sampler._study.optimal_trials())[0]`.



<details>
<summary>🧹 Proposed fix</summary>

```diff
-    best = list(sampler._study.optimal_trials())[0].materialize()
+    best = next(iter(sampler._study.optimal_trials())).materialize()

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/profiler/spica/tests/test_sampler.py` at line 274,
Update the best-trial selection to use
next(iter(sampler._study.optimal_trials())) instead of materializing the full
collection and indexing its first element, then call materialize() on the
resulting trial.

Source: Linters/SAST tools

components/src/dynamo/profiler/spica/tests/test_evaluator.py (1)

33-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Hardcoded /tmp/t.jsonl trace paths across the Spica test suite. Both files hardcode a /tmp path as Workload/SmartSearchConfig trace_path in shared test fixtures instead of using tmp_path/tempfile; the path is never actually opened (replay calls are mocked), but this repeats the same guideline violation in both files.

• components/src/dynamo/profiler/spica/tests/test_evaluator.py#L33-L34: replace the "/tmp/t.jsonl" literal in _wl() (and the other 7 occurrences at Lines 86, 152, 255, 273, 279, 469, 483) with a tmp_path-derived path.
• components/src/dynamo/profiler/spica/tests/test_search.py#L21-L37: replace the "/tmp/t.jsonl" literal in _config() (and the other 2 occurrences at Lines 261, 346) with a tmp_path-derived path.

As per path instructions, "Don't hardcode temp paths; use tmp_path/tempfile."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/profiler/spica/tests/test_evaluator.py` around lines 33
- 34, Replace every hardcoded "/tmp/t.jsonl" trace_path in
components/src/dynamo/profiler/spica/tests/test_evaluator.py at lines 33-34, 86,
152, 255, 273, 279, 469, and 483 with paths derived from the relevant pytest
tmp_path fixture, updating _wl() and affected tests as needed. Apply the same
change in components/src/dynamo/profiler/spica/tests/test_search.py at lines
21-37, 261, and 346, including _config(); ensure each fixture or test receives
and uses tmp_path without changing the mocked replay behavior.

Source: Path instructions

components/src/dynamo/profiler/spica/tests/test_packaging.py (1)

13-27: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Prefer TOML parsing here

Use tomllib/tomli instead of regex + ast.literal_eval to read pyproject.toml; this is more robust to formatting changes, and the import needs a 3.10-compatible fallback because the repo’s minimum Python is 3.10.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/profiler/spica/tests/test_packaging.py` around lines 13
- 27, The test test_spica_extra_matches_planner_image_requirements should parse
pyproject.toml with tomllib, falling back to tomli for Python 3.10
compatibility, instead of regex and ast.literal_eval. Read the spica dependency
list from the parsed TOML structure while preserving the existing comparison
with planner_dependencies.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/profiler/spica/kv_load.py`:
- Around line 127-133: Validate expected_tokens_per_request immediately after
calculating it in the KV capacity flow, before the concurrency_capacity
division. Raise the existing InfeasibleKVCapacity domain error when the value is
less than or equal to zero, preserving the current capacity check for positive
values.
- Around line 71-87: Validate block_size immediately after parsing it in the
surrounding capacity calculation, rejecting values less than or equal to zero
before the floor-division in per_rank_usable_tokens. Use the existing function’s
established error-handling convention, and preserve the current calculation for
positive block sizes.

In `@components/src/dynamo/profiler/spica/search.py`:
- Around line 505-531: Replace the `assert` in the `_eval_batch`
result-processing loop with an explicit `if` check that validates both
`candidate` and `observe_metrics` are non-None, then raise a clear runtime
exception when the contract is violated. Preserve the existing sampling, cache,
and `_record` flow for valid results.
- Around line 513-522: Update the failed/infeasible branch in the search
processing flow to increment the appropriate tally counter for every entry in
duplicates, matching the feasible-path cache-hit accounting while retaining
observe_infeasible and failure-reason tracking.

In `@components/src/dynamo/profiler/spica/tests/conftest.py`:
- Around line 17-31: The test classification currently treats real
model-resolution coverage as unit tests. Update _INTEGRATION_TESTS in
conftest.py to include test_model_hw.py and test_kv_estimate.py; in
test_model_hw.py lines 36-54, annotate the affected tests with the appropriate
pytest.mark.model(...) markers; and in test_kv_estimate.py lines 174-211,
annotate only the real-estimate cases with pytest.mark.model(...), leaving
synthetic/unit cases unchanged.
- Around line 17-31: Update pytest_collection_modifyitems so integration tests
in _INTEGRATION_TESTS receive the post_merge marker instead of pre_merge, while
non-integration Spica tests retain pre_merge. Preserve the existing integration,
timeout, gpu_0, and planner markers.

In `@components/src/dynamo/profiler/spica/tests/test_cli.py`:
- Around line 19-25: Add explicit timeouts to every subprocess.run call in
test_cli_help_is_marked_experimental, test_cli_rejects_malformed_yaml, and
test_cli_rejects_invalid_config, and add the required pytest timeout marker to
each test. Use consistent bounded timeout values for both the subprocess calls
and test markers.

In `@components/src/dynamo/profiler/spica/tests/test_replay_integration.py`:
- Around line 148-162: Add a pytest timeout marker directly above
test_smart_search_returns_ranked_candidate, using the repository’s established
timeout value or convention for ProcessPool-driving tests. Keep the test body
and assertions unchanged.

In `@components/src/dynamo/profiler/spica/tests/test_search_space.py`:
- Around line 22-31: Update the _config test helper to stop hardcoding
"/tmp/t.jsonl"; require callers needing a real trace path to provide trace_path,
typically using pytest's tmp_path fixture and passing str(tmp_path / "t.jsonl")
through ss_overrides. Preserve the existing default configuration for all other
fields.

In
`@docs/components/profiler/spica/examples/glm5-disagg-pareto-frontier-sweep.md`:
- Around line 8-16: Update the adjacent [!WARNING] and [!IMPORTANT] blockquotes
in the markdown so the separating blank line is also prefixed with `>`,
preserving them as distinct admonitions for Markdown renderers.

In `@examples/profiler/spica/README.md`:
- Around line 76-77: Update the Spica documentation link in the README to
reference a URL that already exists on the target branch, or add the PR-only
path to the repository’s lychee ignore configuration so link checking does not
flag it.

---

Nitpick comments:
In `@components/src/dynamo/profiler/spica/model_hw.py`:
- Around line 26-32: Remove the direct dependency on the private helpers
_estimate_model_weight_bytes and _get_system_config in the model hardware
profiling flow, replacing them with a supported public AIConfigurator accessor
if available. If no public accessor exists, add a focused compatibility test
covering the required behavior and isolate the private-helper usage behind that
compatibility boundary.

In `@components/src/dynamo/profiler/spica/sampler.py`:
- Around line 98-99: The lazy imports are intentional for optional, heavy
dependencies, but the repeated Vizier imports need consolidation. In
components/src/dynamo/profiler/spica/sampler.py:98-99, retain and document the
deferred Vizier import strategy, then reuse a single local/module-level import
across __init__, _update_projection_metadata, observe, and observe_infeasible.
In components/src/dynamo/profiler/spica/load_predictor_sweep.py:198-212, retain
the lazy dynamo-planner imports and document that deferral is required for Rust
runtime and prophet/pmdarima/filterpy dependency loading; no top-level import is
needed there.

In `@components/src/dynamo/profiler/spica/search.py`:
- Around line 145-157: Add logger.exception calls in both broad Exception
handlers surrounding candidate construction and evaluator.evaluate, using lazy
formatting and including the relevant operation context and exception details
before returning the existing failed results. Preserve the current non-aborting
sweep behavior and return values.

In `@components/src/dynamo/profiler/spica/tests/test_evaluator.py`:
- Around line 33-34: Replace every hardcoded "/tmp/t.jsonl" trace_path in
components/src/dynamo/profiler/spica/tests/test_evaluator.py at lines 33-34, 86,
152, 255, 273, 279, 469, and 483 with paths derived from the relevant pytest
tmp_path fixture, updating _wl() and affected tests as needed. Apply the same
change in components/src/dynamo/profiler/spica/tests/test_search.py at lines
21-37, 261, and 346, including _config(); ensure each fixture or test receives
and uses tmp_path without changing the mocked replay behavior.

In `@components/src/dynamo/profiler/spica/tests/test_packaging.py`:
- Around line 13-27: The test
test_spica_extra_matches_planner_image_requirements should parse pyproject.toml
with tomllib, falling back to tomli for Python 3.10 compatibility, instead of
regex and ast.literal_eval. Read the spica dependency list from the parsed TOML
structure while preserving the existing comparison with planner_dependencies.

In `@components/src/dynamo/profiler/spica/tests/test_sampler.py`:
- Line 274: Update the best-trial selection to use
next(iter(sampler._study.optimal_trials())) instead of materializing the full
collection and indexing its first element, then call materialize() on the
resulting trial.

In `@docs/components/profiler/spica/traffic.md`:
- Around line 114-116: Add the text language identifier to the fenced code block
documenting resolved_request_count, changing the opening fence to use text while
preserving the existing formula and closing fence.

In `@lib/bindings/python/rust/llm/aic_callback.rs`:
- Around line 32-39: Remove the redundant unsafe Send and Sync implementations
for PyAicCallback; its session: Py<PyAny> field already provides these auto
traits. Leave the struct definition and Python GIL access behavior unchanged.

In `@lib/bindings/python/tests/replay/test_replay_aic_capacity.py`:
- Around line 203-207: Update the match pattern in the pytest.raises assertion
surrounding _run_direct_aic_replay to use a raw string literal, preserving the
existing regex and expected exception behavior.

In `@pyproject.toml`:
- Around line 90-102: The dependency versions are consistent; no code change is
required. Monitor the exact tfp-nightly pin in the spica dependency set and
replace it with a stable tensorflow-probability release when the google-vizier
constraint permits.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 98b58e38-c90f-419a-abc7-dd607b8e70ad

📥 Commits

Reviewing files that changed from the base of the PR and between 59dd504 and 22f608e.

⛔ Files ignored due to path filters (1)

• docs/assets/img/spica-glm5-disagg-pareto-frontier-sweep.png is excluded by !**/*.png

📒 Files selected for processing (65)

• components/src/dynamo/profiler/spica/__init__.py
• components/src/dynamo/profiler/spica/__main__.py
• components/src/dynamo/profiler/spica/_quiet.py
• components/src/dynamo/profiler/spica/config.py
• components/src/dynamo/profiler/spica/deploy.py
• components/src/dynamo/profiler/spica/evaluator.py
• components/src/dynamo/profiler/spica/kv_estimate.py
• components/src/dynamo/profiler/spica/kv_load.py
• components/src/dynamo/profiler/spica/load_predictor_sweep.py
• components/src/dynamo/profiler/spica/model_hw.py
• components/src/dynamo/profiler/spica/parallel_enum.py
• components/src/dynamo/profiler/spica/parallel_projection.py
• components/src/dynamo/profiler/spica/planner.py
• components/src/dynamo/profiler/spica/sample.py
• components/src/dynamo/profiler/spica/sampler.py
• components/src/dynamo/profiler/spica/score.py
• components/src/dynamo/profiler/spica/search.py
• components/src/dynamo/profiler/spica/search_space.py
• components/src/dynamo/profiler/spica/tests/conftest.py
• components/src/dynamo/profiler/spica/tests/data/mooncake_tiny.jsonl
• components/src/dynamo/profiler/spica/tests/data/spica_native_features.yaml
• components/src/dynamo/profiler/spica/tests/test_cli.py
• components/src/dynamo/profiler/spica/tests/test_config.py
• components/src/dynamo/profiler/spica/tests/test_deploy.py
• components/src/dynamo/profiler/spica/tests/test_evaluator.py
• components/src/dynamo/profiler/spica/tests/test_kv_estimate.py
• components/src/dynamo/profiler/spica/tests/test_kv_load.py
• components/src/dynamo/profiler/spica/tests/test_load_predictor_sweep.py
• components/src/dynamo/profiler/spica/tests/test_load_predictor_sweep_integration.py
• components/src/dynamo/profiler/spica/tests/test_model_hw.py
• components/src/dynamo/profiler/spica/tests/test_packaging.py
• components/src/dynamo/profiler/spica/tests/test_parallel_enum.py
• components/src/dynamo/profiler/spica/tests/test_parallel_projection.py
• components/src/dynamo/profiler/spica/tests/test_planner.py
• components/src/dynamo/profiler/spica/tests/test_quiet.py
• components/src/dynamo/profiler/spica/tests/test_replay_integration.py
• components/src/dynamo/profiler/spica/tests/test_sample.py
• components/src/dynamo/profiler/spica/tests/test_sampler.py
• components/src/dynamo/profiler/spica/tests/test_score.py
• components/src/dynamo/profiler/spica/tests/test_search.py
• components/src/dynamo/profiler/spica/tests/test_search_space.py
• components/src/dynamo/replay/main.py
• container/deps/requirements.planner.txt
• container/templates/wheel_builder.Dockerfile
• docs/components/profiler/README.md
• docs/components/profiler/spica/README.md
• docs/components/profiler/spica/examples/glm5-disagg-pareto-frontier-sweep.md
• docs/components/profiler/spica/examples/planner-goodput-per-gpu-sweep.md
• docs/components/profiler/spica/examples/router-e2e-latency-sweep.md
• docs/components/profiler/spica/optimization-goal.md
• docs/components/profiler/spica/overview.md
• docs/components/profiler/spica/sample.md
• docs/components/profiler/spica/search-space.md
• docs/components/profiler/spica/traffic.md
• docs/index.yml
• examples/profiler/spica/README.md
• examples/profiler/spica/configs/glm5-disagg-pareto-frontier.yaml
• examples/profiler/spica/configs/smart_sweep.yaml
• examples/profiler/spica/tools/gen_sine_trace.py
• examples/profiler/spica/tools/run_load_predictor_sweep.py
• lib/bindings/python/rust/llm/aic_callback.rs
• lib/bindings/python/rust/llm/replay.rs
• lib/bindings/python/tests/replay/test_replay_aic_capacity.py
• pyproject.toml
• tests/report_pytest_markers.py

[tedzhouhk]

Addressed the remaining CodeRabbit review summary items in bb0f21fa9f (plus the earlier unsafe Send/Sync cleanup already present in d5b299a4b4): isolated and tested the AIC 0.9 private-helper boundary; consolidated and documented deferred Vizier/Planner imports; logged broad worker failures with tracebacks; removed shared /tmp test paths; bounded subprocess and ProcessPool tests; corrected integration/model scheduling; used iterator-based best-trial selection; labeled the text fence; separated adjacent admonitions; and made the regex raw. The packaging TOML nit was superseded by the installed-package contract test and scripts/check_spica_packaging.py. The exact tfp-nightly pin remains intentionally monitored until Vizier permits a stable TFP release. Validation: full Spica suite 246 passed, 6 skipped; pre-commit passed; packaging contract passed; Fern check reported 0 errors; CI filter matching passed.

[tedzhouhk]

/ok to test f261132

[tedzhouhk]

CI infrastructure note: deploy-operator has failed twice on this SHA at the Helm Docker Hub registry login with unauthorized: incorrect username or password. The same failure is affecting concurrent PR workflows (for example, run 29875480844/job/88785629256), while this PR's remaining 71 checks pass. I am monitoring for the repository Docker Hub credentials to recover and will rerun the deploy job and dependent status gate.

[jasonqinzhou]

as discussed, we can merge this first, then have a small PR for libs/mocker updates after dynamo aic version bump.

[nv-anants]

reviewed .github, overall structure and packaging changes - lgtm!

[tedzhouhk]

/ok to test d08260a

[jthomson04]

Approving changes in dynamo operator comments.