[codex] Stream live token usage into desktop context meter

[OmarB97]

Summary

• emit token usage snapshots from the agent loop after provider usage is recorded and bridge them through the gateway as token.usage events
• emit a preflight token.usage snapshot before each API call using the already-computed request-size estimate, so providers that only report usage at completion still move the desktop context meter during the active turn
• consume those events in the desktop session hook so the context meter can move before message.complete
• add focused backend, desktop, and TUI coverage, plus a small spawn stdio typing fix so ui-tui type-checks cleanly

Why

The first live-usage patch handled final/canonical provider usage, but the official macOS app could still sit at 0/131.1k while a turn was running because many providers expose usage only at completion. The new preflight snapshot sends the current request/context estimate at call start, then the final provider usage corrects totals afterward.

Fork / install follow-up

• Fork PR fix(desktop): emit preflight context usage OmarB97/hermes-agent#85 merged the backend preflight patch; its post-merge CI exposed a fork-only CLI flag-hoisting regression.
• Fork PR fix(cli): preserve chat-local provider flags OmarB97/hermes-agent#86 fixed that parser regression and all fork CI checks are green.
• The official macOS app on this machine was rebuilt from fork main at c0c1bcd093eb10249cfd972e9f4146452c4534d6 with a clean install stamp.

Verification

• python3 -m pytest tests/tui_gateway/test_token_usage_events.py
• python3 -m pytest tests/run_agent/test_run_agent.py::TestRunConversation::test_preflight_token_usage_emits_before_api_response tests/run_agent/test_run_agent.py::TestRunConversation::test_stop_finish_reason_returns_response
• python3 -m py_compile agent/conversation_loop.py
• npx vitest run --environment jsdom src/lib/token-usage.test.ts src/app/session/hooks/use-message-stream.test.tsx from apps/desktop
• npm run type-check --workspace ui-tui
• Installed-code gateway canary on macOS: token.usage arrived at ~2.1s and message.complete arrived at ~11.9s for the same turn, proving the desktop can receive usage before end-of-turn.

[OmarB97]

Review: approve with suggestions

Reviewed the diff for live token usage streaming into the desktop/TUI context meter. The data flow is complete and correct: backend _emit_token_usage → status_callback("token_usage", ...) → tui_gateway/server.py intercepts → token.usage event → desktop use-message-stream.ts / TUI createGatewayEventHandler.ts → usage store update.

Verified

• last_prompt_tokens is initialized to 0 in ContextCompressor.__init__ (line 650) — direct access in the post-API emit is safe.
• Preflight emit fires before the first API call with estimate_request_tokens_rough, solving the "0/131k" problem.
• Post-API emit fires after every successful _perform_api_call with actual usage from canonical_usage.
• Both desktop and TUI side have defensive parsing (finiteNumber, nonNegativeNumber, positiveNumber) that rejects NaN/null/malformed values.
• Session-scoped filtering: the desktop handler only updates the usage bar for the active session.

Non-blocking suggestions

1. tui_gateway/server.py: Consider adding isinstance(payload, dict) guard before _emit("token.usage", ...) — prevents forwarding non-dict JSON values (already in the working tree but not in this diff).

2. tests: Consider a test for the post-API-call _emit_token_usage path in test_run_agent.py, to mirror test_preflight_token_usage_emits_before_api_response.

3. Style nit: import json inside the method body in tui_gateway/server.py is unconventional but functional.

Decision

Approve — no blocker-severity findings. The PR achieves the stated goal of streaming live context usage into the desktop/TUI meter.

[OmarB97]

Superseded by #39370, which carries this slice (gateway emission, preflight context estimate, desktop + TUI consumption) rebased on current main. Two micro-slices not carried there (scripts/release.py author-email map; execFileNoThrow.ts SpawnOptions stdio typing fix) — will rehome in a future small PR if still wanted. Closing to reduce reviewer surface.