Skip to content

feat(computer-use): local critical-path report for phase spans (RFC #112639 P0) - #113225

Closed
kvnloo wants to merge 1 commit into
NousResearch:mainfrom
kvnloo:feat/cu-critical-path-report-112639
Closed

kvnloo wants to merge 1 commit into
NousResearch:mainfrom
kvnloo:feat/cu-critical-path-report-112639

Conversation

@kvnloo

@kvnloo kvnloo commented Sep 16, 2026

Copy link
Copy Markdown

What does this PR do?

Adds the analysis half of the P0 instrumentation slice from the computer-use latency wave (design RFC #112639). New tools/computer_use/critical_path.py consumes phase spans keyed by the existing observer/request/tool correlation IDs (session_id / task_id / tool_call_id, as carried by the pre_tool_call / post_tool_call and pre_api_request / post_api_request lifecycle hooks) and reconstructs per-task critical paths locally, with no Relay round trip.

For each task the report produces: end-to-end latency, measured phase time as a sum of non-overlapping critical-path segments, the explained fraction, idle gaps between spans, and avoidable idle (approval_wait + dispatch_lock_wait + gaps). The aggregate view gives p50/p95 E2E with the p50 explained fraction — the exit criterion for the P0 slice. The phase vocabulary matches the recording half so the two compose without overlap.

Related Issue

Related to #112639 (P0 wave slice). Closes the measurement gap behind #109569 (per-turn breakdown discarded from a 10-deep deque), #93312 (hardcoded driver readiness probe), and #63357 (capture/settle latency as a black box). Complements #112778, which records the phase spans — this PR only reads them and never records new spans, so the two do not duplicate.

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Tests (adding or improving test coverage)

Changes Made

  • tools/computer_use/critical_path.py (new): Span / TaskReport / AggregateReport, sweep-based critical-path reconstruction (never double-counts overlapping or envelope spans), nearest-rank p50/p95, render() text summary
  • tests/computer_use/test_cua_perf_knobs.py: deterministic fixtures for AX-only, SOM, vision, aux-vision, and capture-after flows; reconstruction order, avoidable-idle, no-double-count, and aggregate tests

How to Test

  1. scripts/run_tests.sh tests/computer_use/test_cua_perf_knobs.py — 8/8 green (5 new)
  2. scripts/run_tests.sh tests/computer_use/ — 57 passed, 1 skipped
  3. Demo on the fixtures (the red-on-base equivalent):
task t-ax: e2e=3020ms measured=2970ms (98.3%) idle=50ms avoidable=65ms
task t-som: e2e=4170ms measured=4130ms (99.0%) idle=40ms avoidable=40ms
task t-vision: e2e=4250ms measured=4230ms (99.5%) idle=20ms avoidable=1020ms
task t-aux: e2e=2980ms measured=2960ms (99.3%) idle=20ms avoidable=20ms
task t-capafter: e2e=3410ms measured=3370ms (98.8%) idle=40ms avoidable=40ms
computer_use critical path: 5 tasks
  e2e p50=3410ms p95=4250ms
  explained p50=99.0% of e2e as measured phases
  avoidable idle total=1185ms

Checklist

Code

  • Commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this is not a duplicate (complements feat(computer-use): add phase telemetry #112778, does not duplicate)
  • My PR contains only changes related to this fix/feature
  • Focused suites run via scripts/run_tests.sh and all pass
  • I have added tests for my changes
  • Behavior-neutral: no schema change, no model-facing behavior change, no tool execution-path changes

Documentation & Housekeeping

  • Docstrings cover the new module — N/A for README/config/AGENTS changes

Authored with AI assistance (Muse, Meta's Muse Spark) under the contributor's direction.

JoaoMarcos44 added a commit to JoaoMarcos44/hermes-agent that referenced this pull request Sep 16, 2026
@kvnloo
kvnloo force-pushed the feat/cu-critical-path-report-112639 branch 2 times, most recently from b5dd9ad to 6052f78 Compare September 16, 2026 22:10
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets labels Sep 16, 2026
@kvnloo
kvnloo force-pushed the feat/cu-critical-path-report-112639 branch 7 times, most recently from 6d786c7 to 1bd629b Compare September 17, 2026 05:08
@kvnloo
kvnloo force-pushed the feat/cu-critical-path-report-112639 branch 6 times, most recently from cf0f6ec to c7111ac Compare September 17, 2026 12:07
NousResearch#112639 P0)

Consumes phase spans keyed by the existing observer/request/tool correlation
IDs (session_id / task_id / tool_call_id) and reconstructs per-task critical
paths locally: ordered phase segments, idle gaps, avoidable idle
(approval_wait, dispatch_lock_wait), and p50/p95 E2E explained as a sum of
measured phases. Complements NousResearch#112778 (phase span recording) without
duplicating it. Behavior-neutral: no schema change, no model-facing change.

authored with AI assistance (Muse, Meta's Muse Spark) under the contributor's direction; reviewed and approved by the contributor before submission.
@kvnloo
kvnloo force-pushed the feat/cu-critical-path-report-112639 branch from c7111ac to 119a348 Compare September 17, 2026 13:08

kvnloo commented Sep 26, 2026

Copy link
Copy Markdown
Author

Closing this analysis-only carrier. It depends on the separate phase recorder #112778, neither half is on current main, and the performance work since this RFC has been more useful when attached directly to a concrete optimization with a before/after control. We should not make a general critical-path reporting framework a prerequisite for improving one demonstrated hot path.

The span vocabulary / no-double-count report remains useful experiment provenance and can be revived in eval tooling if a current performance question actually needs it.

@kvnloo kvnloo closed this Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants