feat(cli): add native timing and source detail reports

[jasonqinzhou]

Behavior

predict --detail time adds native accumulated prefill/decode and per-operation timing, SOL latency/compute/memory comparisons, and latency/SOL ratios alongside serving latency statistics. --detail source adds operation source tags and executed MoE measurement substitutions. all selects all five sections; --detail-top-n limits tables while JSON stdout and prediction.json retain complete, matching evidence.

Evidence comes through the canonical ForwardPassPerfModel API and is captured only when requested. Missing SOL implementations preserve actual timing and provenance with an explicit reason. Whole-model FPM, fixed/polynomial timing, analytical overlays, and adapters without the native export report unavailable operation evidence.

Review map and contracts

• High risk: public performance-model API and replay evidence boundary. Start with crates/core/src/perfmodel/engine/runtime.rs and fpm/model.rs, then crates/core/src/python.rs, engine/timing.rs, and python/aisimulate/src/aisimulate/detail.py.
• Additive contracts: Rust/Python static_phase_diagnostics, ReplayOutputRequirements.capture_performance_diagnostics, timing/source JSON, and optional TimingOperationEvidence.details. Existing Rust struct literals initialize details: None; the constructor does so automatically. The Python wrapper follows the current parent package location, aisimulate_core.
• No selector preserves the existing report shape. --diagnostics-top-n remains an alias. No performance data or frozen golden values changed.
• Review fixes reject contradictory or unexplained SOL states, preserve known fallback records when combining detailed and legacy evidence, reject empty native diagnostics for nonzero work, reject zero MoE topology before calibration, validate adapter records with field-specific errors, and align native fallback/string validation with the JSON schema. The public API witness uses the renamed crate, and the shipped native type stub includes the new method with a live signature/return contract check.

Validation

Final reviewed head: ebc898ec6aa43181c17b085cbe81287426e452e5. Squash-merged into main as 7ae483e942b0a2593d55e4e30957f94640580441 on 2026-09-19 at 02:23:16 UTC. The live main tip is the merge commit, and its tree exactly matches the tested head (d8af2f0545821430a3e3535d61fc1ff525a5ff1c).

• Trusted exact-head Fast CI: passed.
• Trusted exact-head Full CI: passed, 52 successful jobs and two accepted skips (Cargo Deny selected N/A; release staging is not a PR step).
• Manual complete-matrix Full CI on the same head: 54 successful jobs and one expected staging skip.
• Local validation: 1,503 Rust tests passed, one ignored, on the identical Rust tree; 422 expanded Python checks passed before the last Python-only parent fixes and 287 affected checks passed on the final tree. The native type-stub signature/return contract was rerun on the final head and passed.
• Codex and CodeRabbit reviews found no remaining actionable issue. All review threads were resolved, and applicable approval and CODEOWNERS requirements were accepted at merge.

Modeling boundaries

Rust remains the only estimator. Operation totals represent accumulated scheduled rank-local work, not request latency, critical-path duration, or whole-deployment GPU time. SOL compares the same scheduled shapes and is not scaled by synthetic speedup. Source records are native classifications and executed substitutions, not full file/row lineage or an estimator-construction audit. Energy publication behavior is unchanged.

Related: #269 (merged prerequisite), #242 (canonical API), #19 (superseded).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/aisimulate/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 836f4e27-759a-4f2a-889f-9313ddd45245

📥 Commits

Reviewing files that changed from the base of the PR and between c45da2c and ef9d16f.

📒 Files selected for processing (9)

• crates/core/src/engine/timing.rs
• crates/core/src/perfmodel/engine/runtime.rs
• crates/core/src/perfmodel/fpm/tests.rs
• crates/core/src/perfmodel/py.rs
• crates/core/src/python.rs
• python/aisimulate/src/aisimulate/runner.py
• python/aisimulate/src/aisimulate_core/sdk/rust_engine_step.py
• python/aisimulate/tests/cross_package/test_core_public_api.py
• tests/test_unified_traffic_runtime.py

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• ai-dynamo/dynamo (manual)
• ai-dynamo/aiconfigurator (manual)

Included review availability: Your plan provides up to 12 included reviews per hour; 8 remain after this review.

📜 Recent review details

🧰 Additional context used

📓 Path-based instructions (9)

Preserve the Rust single oracle: Python may describe operations, load raw data, orchestrate, and present results, but must not compute per-op performance values.

⚙️ CodeRabbit configuration file

Files:

• python/aisimulate/src/aisimulate_core/sdk/rust_engine_step.py

Check unified CLI, Replay, Sweeper, and orchestration behavior together.

⚙️ CodeRabbit configuration file

Files:

• python/aisimulate/src/aisimulate/runner.py

Enforce the single-oracle and golden-diff rules in python/aisimulate/.claude/rules/rust-core/parity.md.

⚙️ CodeRabbit configuration file

Files:

• crates/core/src/perfmodel/fpm/tests.rs
• crates/core/src/perfmodel/py.rs
• crates/core/src/perfmodel/engine/runtime.rs

Review scheduler state transitions and event ordering, including cancel, preempt, drain, retry, duplicate, empty, and terminal paths.

⚙️ CodeRabbit configuration file

Files:

• crates/core/src/engine/timing.rs

Treat top-level exports and bindings as public and release boundaries.

⚙️ CodeRabbit configuration file

Files:

• crates/core/src/python.rs

Require coverage of the changed behavior and its negative or boundary cases.

⚙️ CodeRabbit configuration file

Files:

• tests/test_unified_traffic_runtime.py

Read REVIEW.md before commenting.

⚙️ CodeRabbit configuration file

Files:

• tests/test_unified_traffic_runtime.py
• crates/core/src/perfmodel/fpm/tests.rs
• python/aisimulate/src/aisimulate/runner.py
• python/aisimulate/tests/cross_package/test_core_public_api.py
• crates/core/src/engine/timing.rs
• python/aisimulate/src/aisimulate_core/sdk/rust_engine_step.py
• crates/core/src/perfmodel/py.rs
• crates/core/src/perfmodel/engine/runtime.rs
• crates/core/src/python.rs

Do not reintroduce them.

📄 CodeRabbit inference engine (python/aisimulate/.claude/rules/rust-core/parity.md)

Files:

• crates/core/src/perfmodel/fpm/tests.rs
• python/aisimulate/src/aisimulate_core/sdk/rust_engine_step.py
• crates/core/src/perfmodel/py.rs
• crates/core/src/perfmodel/engine/runtime.rs

Before making any change under: `python/aisimulate/src/aiconfigurator/generator/**` MUST read: `python/aisimulate/.claude/rules/generator-development.md` Before making any change under `python/aisimulate/collector/**` MUST read: `python/ais...

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• tests/test_unified_traffic_runtime.py
• crates/core/src/perfmodel/fpm/tests.rs
• python/aisimulate/src/aisimulate/runner.py
• python/aisimulate/tests/cross_package/test_core_public_api.py
• crates/core/src/engine/timing.rs
• python/aisimulate/src/aisimulate_core/sdk/rust_engine_step.py
• crates/core/src/perfmodel/py.rs
• crates/core/src/perfmodel/engine/runtime.rs
• crates/core/src/python.rs

🪛 ast-grep (0.45.3)

python/aisimulate/src/aisimulate/runner.py

[info] 426-430: use jsonify instead of json.dumps for JSON output
Context: json.dumps(
execution_spec,
allow_nan=False,
separators=(",", ":"),
)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

python/aisimulate/tests/cross_package/test_core_public_api.py

[info] 373-383: use jsonify instead of json.dumps for JSON output
Context: json.dumps(
{
"model": "Qwen/Qwen3-32B",
"system": "h200_sxm",
"backend": "vllm",
"backend_version": "0.24.0",
"worker_type": "aggregated",
"estimation_mode": "op_level",
"tp": 2,
}
)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🔀 Multi-repo context ai-dynamo/dynamo, ai-dynamo/aiconfigurator

Linked repositories findings

ai-dynamo/dynamo

• DynamoReplayRunner accepts ReplayOutputRequirements but does not forward capture_performance_diagnostics into replay calls. components/src/dynamo/replay/simulation.py:101-128 [::ai-dynamo/dynamo::]
• Public trace and synthetic replay functions expose no diagnostics-capture parameter. components/src/dynamo/replay/api.py:133-164,299-332 [::ai-dynamo/dynamo::]
• Dynamo pins aisimulate==0.12.0, so consuming this feature requires a coordinated dependency/API update. pyproject.toml:17; container/deps/requirements.aisimulate.txt:5 [::ai-dynamo/dynamo::]

ai-dynamo/aiconfigurator

• AIC is now a compatibility/migration repository; its documentation directs new features and integrations to AISimulate, and it has no references to the new diagnostics APIs. README.md:40-41; CONTRIBUTING.md:11-23 [::ai-dynamo/aiconfigurator::]

🔇 Additional comments (9)

crates/core/src/engine/timing.rs (1)

72-72: LGTM!

Also applies to: 92-99, 124-161, 180-212, 682-820, 874-874

crates/core/src/perfmodel/py.rs (1)

1783-1800: LGTM!

python/aisimulate/src/aisimulate_core/sdk/rust_engine_step.py (1)

312-322: LGTM!

python/aisimulate/tests/cross_package/test_core_public_api.py (1)

353-389: LGTM!

crates/core/src/python.rs (1)

42-42: LGTM!

Also applies to: 393-393, 442-442, 467-489, 793-793, 805-808, 1300-1338, 1513-1514, 1551-1558, 1603-1603, 1665-1671, 1750-1753, 2126-2126, 2603-2603, 2624-2624, 2634-2701, 2900-2956

crates/core/src/perfmodel/engine/runtime.rs (1)

1337-1484: LGTM!

Also applies to: 2472-2482, 2493-2536, 2676-2707

crates/core/src/perfmodel/fpm/tests.rs (1)

1966-1995: LGTM!

python/aisimulate/src/aisimulate/runner.py (1)

423-426: LGTM!

Also applies to: 465-465

tests/test_unified_traffic_runtime.py (1)

796-796: LGTM!

Also applies to: 810-826, 854-858

---

📝 Summary

Risk level: Medium

Three areas need human attention:

1. Diagnostic schema and compatibility

   • Verify new time and source payloads against legacy payloads.
   • Confirm malformed system YAML fails explicitly.

2. Native and fallback parity

   • Verify static_phase_diagnostics across Rust, PyO3, Python, replay, and CLI paths.
   • Check phase coordinates, prefix handling, MTP sizing, and latency ratios.

3. Evidence retention

   • Confirm --detail-top-n limits tables only.
   • Confirm JSON retains all operations and fallback reasons.

Changed behavior and public contracts

• predict --detail time reports phase timing, operation timing, SOL comparisons, and latency/SOL ratios.
• predict --detail source reports operation provenance and executed MoE substitutions.
• --detail all includes source diagnostics.
• --detail-top-n aliases --diagnostics-top-n.
• New diagnostics models and static_phase_diagnostics APIs expose timing, energy, source, SOL, and fallback data.
• Performance-diagnostics capture remains opt-in.
• Legacy report shapes remain unchanged without a detail selector.

Technical quality

The supplied evidence reports 1,503 Rust tests and 422 Python checks. Review severity counts are unavailable because no current finding list was supplied. The shell results did not provide additional diff or test evidence.

Merge readiness

Merge readiness is not established. Exact-head Fast CI was pending, and Full CI was deferred pending parent change #269. Confirm the arm64 YAML test update, target-platform results, and native-versus-fallback report parity before merge.

Walkthrough

Adds end-to-end static operation diagnostics from the Rust engine through replay and the Python CLI. It defines SOL and fallback evidence, validates and accumulates timing details, exposes time and source sections, and updates schemas, tests, and documentation.

Changes

Static performance diagnostics

Layer / File(s)	Summary
Diagnostic contracts and evidence accumulation crates/core/src/perfmodel/engine/diagnostics.rs, crates/core/src/engine/timing.rs	Adds serializable diagnostic models. Timing evidence validates SOL and fallback data, accumulates compatible metrics, and deduplicates fallbacks.
Native phase diagnostics API crates/core/src/perfmodel/engine/*, crates/core/src/perfmodel/fpm/*, crates/core/src/perfmodel/py.rs, python/aisimulate/src/aisimulate_core/*, crates/core/src/perfmodel/operators/moe_a2a.rs	Adds static prefill/decode diagnostics, Rust and Python bindings, native-model restrictions, topology validation, and related tests.
Replay capture and report aggregation crates/core/src/python.rs, python/aisimulate/src/aisimulate/sweeper/replay.py, crates/tests/public-api/src/lib.rs	Propagates the capture flag through execution, retains native diagnostic models, and reports accumulated latency, SOL, provenance, fallback, and availability data.
CLI capture, validation, and rendering python/aisimulate/src/aisimulate/{main.py,runner.py,detail.py,cli_args.py}, tests/test_unified_cli.py, tests/test_unified_traffic_runtime.py	Adds source and performance-diagnostic output, validates nested diagnostic data, limits formatted tables, preserves full JSON output, and covers unavailable evidence.
Schema and diagnostic documentation docs/cli/prediction-details.schema.json, docs/cli/*.md, docs/core-api.md	Defines diagnostic statuses, SOL metrics, fallback records, source and time structures, selector behavior, and unsupported-provider behavior.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~120 minutes

🚥 Pre-merge checks | ✅ 8

✅ Passed checks (8 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cross-Layer Contract	✅ Passed	No stale cross-layer contract found. The diff traces native diagnostics from public Rust models and pub mod diagnostics, through the PyO3 JSON method, the Python SDK wrapper and _native.pyi, into …
Modeling And Data Evidence	✅ Passed	The PR supplies provenance and reproducible evidence for the new model outputs. StaticOperationDiagnostics carries operation source and executed fallback records, and CLI tests verify that source …
Compatibility Boundaries	✅ Passed	The reviewed diff keeps the compatibility boundaries synchronized. The Rust diagnostic API is reachable through the existing aisimulate-core crate, and the PyO3 method, _native.pyi signature, and …
Review Evidence	✅ Passed	The description names the relevant CLI commands (predict --detail time, source, all, and --detail-top-n), reports local Rust and Python test results, and lists negative or boundary checks for …
Title check	✅ Passed	The title precisely describes the main behavioral change: adding native timing and source detail reports to the CLI.
Description check	✅ Passed	The description is complete and directly addresses the behavior change, review risks, public and serialized contracts, validation evidence, modeling boundaries, and related work. It uses different hea…

---

_{Comment @coderabbitai help to get the list of available commands.}

[jasonqinzhou]

@coderabbitai full review

[coderabbitai]

⚠️ Action not completed

Head commit changed.

[jasonqinzhou]

@coderabbitai review

[jasonqinzhou]

The parent branch advanced to 7301939 with the DeepSeek V4.1 changes from #159. This introduces a content conflict in crates/core/src/python.rs at the native replay boundary. I am integrating the new parent and will retain both its architecture-specific timing/storage behavior and this PR’s opt-in diagnostic capture, then refresh validation on the resulting head.

[jasonqinzhou]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Pull request base or head changed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jasonqinzhou]

Codex fresh self-review of 4cb1eae2a7059b5c8aade2e7ef1ed07fef558faa against current parent 7301939e0efc2c919b090102306240b9c64789e1 found no remaining actionable issue. The parent introduces DeepSeek V4.1 support and changed the native replay binding. I resolved all three conflicts by preserving both diagnostic_model and decoder_replay, then checked the submitted diff and the affected canonical-model, phase-query, decode-coordinate, prefill-validation, and report paths.

A new focused regression proves that diagnostics and SOL retain the parent's bounded decoder-prefill geometry for queries around its window boundary, while decode work remains unchanged. It uses independent memory-probe token counts. All 1,503 Rust library tests passed (1 ignored), including this regression; the native extension rebuilt successfully. The expanded Python CLI/replay/AFD/estimator/API/SDK/DeepSeek-bridge validation is running.

All earlier bot/self findings remain fixed and resolved; the human stub finding remains fixed and answered for owner closure. Current-head Fast CI and CodeRabbit review precede the next Full CI dispatch. Earlier successful CI runs are historical evidence and do not replace validation after this parent integration.

[jasonqinzhou]

@coderabbitai review

[jasonqinzhou]

Codex fresh self-review of ef9d16f55d5b4cac18b6f5830ce4059baa00ef89 against parent 6eab22fc8a2c38fd941d98344e98318be6c77c29 found no remaining actionable issue. I re-read the submitted timing/source diff and public/native/adapter/schema/documentation boundaries, and reviewed the latest parent changes for decoder-replay validation. This merge changes only four inherited Python files; the Rust tree is identical to the one with 1,503 passing tests (one ignored), and the native extension remains current.

The expanded preceding Python run passed 422 tests. On this head, all 286 affected runner, DeepSeek bridge, and CLI checks pass, together with Ruff checks and formatting. Existing bot/self threads are fixed and resolved. The human type-stub thread is fixed and answered, awaiting its owner's re-review/closure.

Exact-head Fast CI and CodeRabbit review are pending. Full CI is intentionally deferred until parent #269 settles, per the requested sequencing. This is a review checkpoint, not a readiness or merge verdict.

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jasonqinzhou]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jasonqinzhou]

Codex fresh integration review of 92866bf7917444269539af99fa6e28e16819b1b1 against parent 5735cd22d21f329f42eefcdca43d05e36c359ed1: no remaining actionable finding. Parent #269 completed Full CI successfully at this base, so its final lightweight-import fix is now integrated. The complete feature diff against the parent is byte-identical to the previously reviewed diff (SHA-256 e657bf480b3baa32004685915d88b813a09469b5d81f8ec0721b882ca745f45b).

I rechecked the inherited change and the affected configuration, supervision, decoder-replay validation and CLI boundaries. Importing supervision/argument/output setup remains lightweight; requesting decoder replay still validates the canonical model/backend. All 287 affected Python tests pass, including the subprocess import-contract test, together with Ruff lint/format and diff hygiene. Rust source is unchanged from the 1,503-pass, one-ignored run and rebuilt native extension.

Existing bot/self findings remain fixed and resolved. The human stub thread remains fixed and answered for owner closure. Exact-head Fast CI and CodeRabbit review are requested. Full CI will follow their completion; this is not yet a readiness verdict.

[jasonqinzhou]

PR #269 has squash-merged as 7cfa4f20d9cda458eaa1febad7e82a488784518b, and GitHub retargeted this PR to main. GitHub now reports an ancestry conflict, despite main having exactly the same tree as the parent already integrated into #272 (5235bb89f3815efa5ac4b25b6700eb7cb750a66e).

I will record the squash commit in this branch's ancestry while preserving the complete tested tree, verify that the timing/source diff remains identical, and refresh exact-head review and CI. Full CI on the preceding 92866bf7 head passed 54 jobs with one expected staging skip. This ancestry repair changes no files.

[jasonqinzhou]

Codex exact-head ancestry/integration review of ebc898ec6aa43181c17b085cbe81287426e452e5 against main 7cfa4f20d9cda458eaa1febad7e82a488784518b: no remaining actionable finding. The main squash commit has the exact tree of the already integrated parent. Recording its ancestry preserves the complete tested tree (d8af2f0545821430a3e3535d61fc1ff525a5ff1c) and the complete 25-file feature diff (SHA-256 e657bf480b3baa32004685915d88b813a09469b5d81f8ec0721b882ca745f45b). Current main is now an ancestor of this head.

The preserved tree passed Full CI 35412879730 (54 successful jobs, one expected release-staging skip), 287 affected Python checks, and the previously recorded native/expanded Python validation. The live native method/type-stub contract test was rerun on this head and passed. No code or output behavior changed in this ancestry repair. Exact-head Fast CI and review are being refreshed before Full CI; historical checks are not substituted for required current-head checks.

The human type-stub finding is fixed in the shipped native stub and covered by the live signature/return regression. I verified normal platform permission to resolve that addressed thread; all applicable human/CODEOWNER approval requirements remain in force. This review is not an approval.

[jasonqinzhou]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jasonqinzhou]

/ok to test ebc898e

[jasonqinzhou]

Merged and verified: tested head ebc898ec6aa43181c17b085cbe81287426e452e5 became squash commit 7ae483e942b0a2593d55e4e30957f94640580441 on main at 2026-09-19 02:23:16 UTC. The live main tip is that commit, and its tree is exactly identical to the tested head: d8af2f0545821430a3e3535d61fc1ff525a5ff1c.

Trusted Fast CI 35414892097 and Full CI 35414892260 passed on the exact head. The manual complete-matrix Full CI 35413948520 also passed. All review threads are resolved and required approval/CODEOWNERS gates were accepted. The description now records the final validation evidence, and the merge guard has been stopped.