Skip to content

fix(cli): fix gpu command to restore stderr output - #844

Merged
ndizazzo merged 4 commits into
mainfrom
fix/gpu-command-alias-output
Jun 14, 2026
Merged

fix(cli): fix gpu command to restore stderr output#844
ndizazzo merged 4 commits into
mainfrom
fix/gpu-command-alias-output

Conversation

@ndizazzo

@ndizazzo ndizazzo commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

GPU inspection and benchmark-refresh failures now tell operators what went wrong instead of exiting silently. The gpu and gpus command spellings are locked as exact synonyms, the hidden GPU benchmark path reports backend/build problems on stderr, and CUDA/ROCm release builds now include the GPU benchmark backend features they advertise at runtime.

This also updates the testing docs to use the supported benchmark-refresh command: mesh-llm gpus detect --json.

What changed

  • mesh-llm gpu detect and mesh-llm gpus detect are explicitly covered as the same command path.
  • Early command failures now emit visible stderr before returning a non-zero exit code.
  • The hidden benchmark command used by GPU fingerprint refresh now surfaces missing backend support instead of failing with no output.
  • Parent GPU fingerprint refresh failures now include the benchmark child exit status when the child exits with empty stderr.
  • Linux CUDA/ROCm release builds now carry the matching GPU benchmark backend feature flags.
  • Public testing docs no longer point users at the unsupported mesh-llm gpu benchmark command.

Before / After

Invalid old GPU benchmark command

Before, this was part of the confusion because stale docs referenced it, and related command failures could be hard to interpret.

$ mesh-llm gpu benchmark
# non-zero exit, often no useful operator-facing output in nearby failure paths

After, benchmark remains unsupported under gpu/gpus, but the CLI prints a normal error and usage text.

$ mesh-llm gpu benchmark
error: unrecognized subcommand 'benchmark'

Usage: mesh-llm gpus [OPTIONS] [COMMAND]

Supported GPU benchmark refresh

Before, users could reasonably try the stale documented command and miss the supported refresh path.

$ mesh-llm gpu benchmark --json | jq .

After, docs and tests use the supported command.

$ mesh-llm gpus detect --json | jq .

Both spellings remain valid synonyms:

$ mesh-llm gpu detect --help
$ mesh-llm gpus detect --help

Missing CUDA benchmark support

Before, Jetson/aarch64 hosts could select CUDA for fingerprinting, then the hidden benchmark command could exit 1 with no stdout or stderr.

$ mesh-llm benchmark run-gpu --backend cuda
# exit 1, no output

After, the same failure is visible to the operator.

$ mesh-llm benchmark run-gpu --backend cuda
CUDA benchmark backend was not compiled into this mesh-llm binary

Empty-stderr benchmark child failures

Before, a failing benchmark child with no stderr made the parent refresh path difficult to diagnose.

mesh-llm gpus detect
# exit 1, no useful explanation

After, the parent reports the child process status so the failure is actionable.

benchmark child exited with status <status>

Validation

  • cargo test -p mesh-llm-cli gpu_and_gpus_spellings_are_synonymous
  • cargo test -p mesh-llm --test cli_errors
  • cargo test -p mesh-llm-system benchmark::tests::test_run_and_save
  • python3 -m unittest scripts.tests.test_build_release
  • just website-build
  • cargo fmt --all -- --check
  • cargo check -p mesh-llm
  • cargo clippy -p mesh-llm-system --all-targets -- -D warnings
  • cargo clippy -p mesh-llm --all-targets -- -D warnings

Summary by CodeRabbit

  • Bug Fixes

    • More reliable fatal-error/emergency output so errors are shown even when normal output paths fail.
  • Refactor

    • Moved fatal-event handling into a dedicated output submodule for clearer behavior and fallbacks.
  • Documentation

    • Clarified local GPU inspection: machine-readable GPU inventory and refreshed benchmark fingerprint command.
  • Tests

    • Consolidated CLI GPU parsing tests; added CLI integration tests for visible stderr on early failures and GPU benchmarks.
    • Added unit tests for benchmark child failure handling and for fatal-output emergency fallback behavior.
  • Chores

    • Release build script preserves dynamic runtime and includes backend-specific GPU benchmark features.
    • CI smoke script: increased configurable startup timeout for dense chain tests.

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Consolidates GPU CLI parsing tests; adds CLI integration and benchmark-subprocess tests; extracts TUI fatal emission into a new fatal module with emergency fallback and tests; fixes build-release feature assembly with validation tests; updates docs and makes a CI timeout configurable.

Changes

Error Visibility and CLI Integration

Layer / File(s) Summary
CLI parsing tests consolidation for GPU commands
crates/mesh-llm-cli/src/parser.rs
Consolidated multiple discrete GPU-related CLI parsing tests into a single parameterized test that verifies gpus and gpu spelling synonymy, --json flag handling, and detect subcommand behavior.
CLI integration tests for error visibility
crates/mesh-llm/tests/cli_errors.rs
Added two integration tests that assert early command failures (auth revoke-node) and GPU benchmark invocation (benchmark run-gpu --backend cuda) exit non-zero, produce no stdout, and include expected fatal or backend-related error messages on stderr.
Benchmark subprocess error handling test
crates/mesh-llm-system/src/benchmark.rs
Added a unit test that spawns a failing benchmark child with empty stderr and verifies run_benchmark_subprocess reports the child-exit status text in its error.
TUI fatal submodule and re-exports
crates/mesh-llm-tui/src/output/mod.rs
Moved fatal-event construction and emission into a new output::fatal module and re-exported emit_fatal_error/emit_fatal_panic from mod.rs, removing prior in-file implementations.
Fatal emission helpers and emergency fallback
crates/mesh-llm-tui/src/output/fatal.rs
Adds mapping from anyhow::Error to OutputEvent::Fatal, emit_fatal_error, emit_fatal_panic, and tests for emergency fallback when normal emission is unavailable or fails.
Testing documentation update for GPU detection
website/src/docs/pages/testing.md
Replaced prior GPU verification command with gpus detect --json and clarified that gpus --json is inventory output while gpus detect --json refreshes benchmark fingerprints and emits automation-oriented payloads.

Build Release GPU Benchmark Feature Flags

Layer / File(s) Summary
Build script feature flag assembly
scripts/build-release.sh
Reworked build-release.sh to initialize cargo_features as an empty array and append dynamic-native-runtime when enabled, then append backend-specific GPU benchmark features (gpu-bench-cuda for CUDA, gpu-bench-hip for ROCm) rather than overwriting the feature list.
Build release script test validation
scripts/tests/test_build_release.py
Added a Python unittest module that stubs required tools, runs the release script under controlled env vars, captures the cargo invocation, and asserts backend-specific GPU benchmark features and dynamic-native-runtime are passed to cargo.

Docs and CI Smoke Script

Layer / File(s) Summary
Testing docs: GPU detect usage
website/src/docs/pages/testing.md
Documentation clarifies gpus --json is inventory output and gpus detect --json refreshes benchmark fingerprints and emits automation payloads.
Configurable dense chain startup timeout
scripts/skippy-ci-smoke.sh
Adds DENSE_CHAIN_STARTUP_TIMEOUT_SECS (default 180) and replaces the previous hardcoded startup timeout for the dense chain smoke test.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • Mesh-LLM/mesh-llm#827: Refactors TUI fatal-event emission into a dedicated module and introduces first-class OutputEvent::Fatal handling.
  • Mesh-LLM/mesh-llm#849: Also adds DENSE_CHAIN_STARTUP_TIMEOUT_SECS to the skippy CI smoke script and uses it for the dense chain startup timeout.

Suggested reviewers

  • michaelneale
  • i386
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly addresses the main objective—fixing the GPU command to restore stderr output so failures are visible to operators instead of failing silently.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/gpu-command-alias-output

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh-llm-tui/src/output/mod.rs`:
- Around line 8498-8501: The current branch only checks
GLOBAL_OUTPUT_MANAGER.get() and calls emit_event but doesn't handle emit_event
failures, so fatal output can be dropped if the manager exists but its
worker/channel is down; change the call site to attempt emit_event and if it
returns or throws an error (or otherwise indicates failure) fall back to
write_emergency_event(&event); specifically, wrap the emit_event invocation in
error handling (or check its Result/Option) and call write_emergency_event when
emit_event fails, ensuring GLOBAL_OUTPUT_MANAGER, emit_event, and
write_emergency_event are the referenced symbols you update.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1a82697e-d3e4-4937-99e2-6f5b7e961a7f

📥 Commits

Reviewing files that changed from the base of the PR and between 6bd7d77 and 601e344.

📒 Files selected for processing (7)
  • crates/mesh-llm-cli/src/parser.rs
  • crates/mesh-llm-system/src/benchmark.rs
  • crates/mesh-llm-tui/src/output/mod.rs
  • crates/mesh-llm/tests/cli_errors.rs
  • scripts/build-release.sh
  • scripts/tests/test_build_release.py
  • website/src/docs/pages/testing.md

Comment thread crates/mesh-llm-tui/src/output/mod.rs Outdated
@ndizazzo ndizazzo self-assigned this Jun 13, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh-llm-tui/src/output/mod.rs`:
- Around line 8496-8510: emit_event_or_write_emergency currently treats
emit(event) returning Ok(()) as success even when GLOBAL_OUTPUT_MANAGER was
never initialized, so change emit_event_or_write_emergency to explicitly detect
the "no global manager" path and fall back to write_emergency in that case: call
emit(event) and if it returns Err use write_emergency as today, but if it
returns Ok then verify the global manager is actually present (e.g., check
GLOBAL_OUTPUT_MANAGER or expose/consume an is_initialized() accessor from the
output manager module) and if not present call write_emergency(&event) and
return its result; update emit_fatal_error to continue using
build_fatal_error_event, emit_event and write_emergency_event unchanged. Also
add a regression test next to the existing BrokenPipe test that ensures
emit_fatal_error(...) produces the emergency/stderr output when
GLOBAL_OUTPUT_MANAGER has not been initialized (i.e., arrange no global manager,
call emit_fatal_error, assert emergency output seen).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 14db0c87-28a6-45e9-8695-0cc595ba0b47

📥 Commits

Reviewing files that changed from the base of the PR and between 601e344 and a1b1614.

📒 Files selected for processing (2)
  • crates/mesh-llm-tui/src/output/mod.rs
  • scripts/skippy-ci-smoke.sh

Comment thread crates/mesh-llm-tui/src/output/mod.rs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/mesh-llm-tui/src/output/mod.rs (1)

8496-8519: 🛠️ Refactor suggestion | 🟠 Major | 🏗️ Heavy lift

Extract fatal/emergency output routing into its own named module.

This change adds a distinct responsibility around fatal-event fallback handling inside an already oversized output/mod.rs. Moving this slice into a focused module such as fatal_output would keep ownership clearer and stop growing a file that is already far past the repo threshold. As per coding guidelines, "When touching a source file that is already over 1,000 lines, first check whether the change adds or exposes a separable responsibility. If it does, split that responsibility into a semantically named module as part of the change, and keep the new file under 1,000 lines."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh-llm-tui/src/output/mod.rs` around lines 8496 - 8519, Split the
fatal/emergency output routing into a dedicated module (e.g., fatal_output) by
moving the functions emit_fatal_error, global_output_manager_initialized, and
emit_event_or_write_emergency (and any helpers like build_fatal_error_event,
emit_event, write_emergency_event references) into that new module, mark APIs as
pub(crate) or pub as needed, update imports/uses so callers still call
fatal_output::emit_fatal_error, and keep the implementation identical while
ensuring the new module is declared in the parent with mod fatal_output so the
file size of the original output/mod.rs is reduced and ownership is clear.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh-llm-tui/src/output/mod.rs`:
- Around line 8509-8518: The helper emit_event_or_write_emergency currently
allows a race where emit(event) returns Ok(()) before the output manager is
initialized; fix by evaluating output_manager_initialized() before deciding the
success branch: call let initialized = output_manager_initialized() first, then
call emit(event.clone()) and only treat Ok(()) as success if initialized is
true; otherwise call write_emergency(&event). Update the logic in
emit_event_or_write_emergency to use the precomputed initialized flag when
matching emit's result so the no-op emit cannot falsely return Ok(()).

---

Outside diff comments:
In `@crates/mesh-llm-tui/src/output/mod.rs`:
- Around line 8496-8519: Split the fatal/emergency output routing into a
dedicated module (e.g., fatal_output) by moving the functions emit_fatal_error,
global_output_manager_initialized, and emit_event_or_write_emergency (and any
helpers like build_fatal_error_event, emit_event, write_emergency_event
references) into that new module, mark APIs as pub(crate) or pub as needed,
update imports/uses so callers still call fatal_output::emit_fatal_error, and
keep the implementation identical while ensuring the new module is declared in
the parent with mod fatal_output so the file size of the original output/mod.rs
is reduced and ownership is clear.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 65e3b7d7-08d5-4b69-8dd5-5d4371e3b2de

📥 Commits

Reviewing files that changed from the base of the PR and between a1b1614 and 74daa9a.

📒 Files selected for processing (1)
  • crates/mesh-llm-tui/src/output/mod.rs

Comment thread crates/mesh-llm-tui/src/output/mod.rs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
scripts/skippy-ci-smoke.sh (1)

47-47: ⚡ Quick win

Remove the duplicate DENSE_CHAIN_STARTUP_TIMEOUT_SECS assignment.

DENSE_CHAIN_STARTUP_TIMEOUT_SECS is already defined at Line 39; redefining it at Line 47 adds config drift risk without functional benefit.

Suggested cleanup
 DENSE_SMOKE_SPLIT_1="${DENSE_SMOKE_SPLIT_1:-}"
 DENSE_SMOKE_SPLIT_2="${DENSE_SMOKE_SPLIT_2:-}"
-DENSE_CHAIN_STARTUP_TIMEOUT_SECS="${DENSE_CHAIN_STARTUP_TIMEOUT_SECS:-180}"
 STAGE_SERVER_BIN="${STAGE_SERVER_BIN:-target/debug/skippy-server}"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/skippy-ci-smoke.sh` at line 47, There is a duplicate assignment of
the environment variable DENSE_CHAIN_STARTUP_TIMEOUT_SECS; remove the redundant
line that reassigns DENSE_CHAIN_STARTUP_TIMEOUT_SECS (leave the original
definition intact) so the script only defines this variable once and avoids
config drift.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/skippy-ci-smoke.sh`:
- Line 47: There is a duplicate assignment of the environment variable
DENSE_CHAIN_STARTUP_TIMEOUT_SECS; remove the redundant line that reassigns
DENSE_CHAIN_STARTUP_TIMEOUT_SECS (leave the original definition intact) so the
script only defines this variable once and avoids config drift.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d4367368-6649-4626-8c33-e077961ed9fa

📥 Commits

Reviewing files that changed from the base of the PR and between 1d53d76 and 0e05573.

📒 Files selected for processing (1)
  • scripts/skippy-ci-smoke.sh

@ndizazzo
ndizazzo force-pushed the fix/gpu-command-alias-output branch from cadd8a0 to 539e7dc Compare June 14, 2026 07:03
@ndizazzo
ndizazzo merged commit f1c6ace into main Jun 14, 2026
32 checks passed
@ndizazzo
ndizazzo deleted the fix/gpu-command-alias-output branch June 14, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant