Skip to content

test(integration): add general live-test tool for the real "default" combo + rootless wire capture - #9744

Closed
hartmark wants to merge 2 commits into
cheaper-inference:release/v3.8.50from
hartmark:test/live-default-combo-tool
Closed

hartmark wants to merge 2 commits into
cheaper-inference:release/v3.8.50from
hartmark:test/live-default-combo-tool

Conversation

@hartmark

@hartmark hartmark commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Two pieces of live-test infrastructure, gated behind explicit env vars so they never run in CI (need podman, real provider credentials, and — for the wire-capture piece — tcpdump/python3):

1. General default-combo live-test tool (RUN_LIVE_WIRE_CAPTURE=1 is for piece 2 below; this piece reuses omniroute-beta-style gating via OMNIROUTE_API_KEY) — tests/integration/liveDefaultComboShared.ts + live-default-combo-workload.test.ts. Unlike the existing liveGeminiShared.ts (which provisions its own narrow 2-model Gemini-only combo), this reads the REAL default combo currently configured on the target instance directly from the DB and exercises every provider/model step in it directly, bypassing combo routing — so live-test coverage always matches whatever the operator actually has configured instead of a hardcoded snapshot that goes stale the moment the combo changes. Live-verified earlier this session: 14/18 models pass consistently across non-streaming + streaming Chat Completions and streaming Responses API against the real 18-model/5-provider default combo; the 4 consistent failures were real external state (a provider temporarily out of credits, one deprecated free-tier model), not code regressions.

2. Rootless wire-capture correlation (RUN_LIVE_WIRE_CAPTURE=1) — tests/integration/liveContainerHarness.ts, wireCapture.ts, live-default-combo-wire-capture.test.ts. Spins up a dedicated, throwaway podman container running the target branch's own code, captures its real wire traffic via podman unshare nsenter --net=<container netns> -- tcpdump (no root needed — verified working live), sends a sample of real requests through the real default combo, and cross-checks each one's app-level JSON result against the actual HTTP status line + TCP behavior observed on the wire via scripts/sre/tcp-close-analyzer.py's stream reassembly (that script's own docstring now documents the rootless capture command alongside the existing root-requiring one). Catches bugs where the app layer claims one thing but the wire shows another — e.g. a truncated/reset stream, or a status mismatch.

Building this surfaced two real, non-obvious bugs along the way (both fixed inline, not part of this PR's own diff since they're general build-tooling issues found via liveContainerHarness.ts needing to actually build the target image):

  • podman unshare -> nsenter -> tcpdump is a 3-level subprocess chain; SIGTERM to the top-level tracked process doesn't reliably reach the tcpdump grandchild, leaving orphaned processes and a truncated/unreadable pcap. Fixed with -U (unbuffered tcpdump writes) + a pkill -f <pcap path> fallback in wireCapture.ts's own cleanup.
  • Wire capture happens inside the container's own network namespace, where only the container's internal listening port is meaningful — filtering on the dynamically-assigned host port (the natural first instinct) silently captures nothing.

⚠️ base-red inherited: #9737

Test plan

  • npm run typecheck:core — clean
  • npm run lint — clean
  • npm run check:file-size — clean
  • Live-verified end-to-end from a clean cherry-pick onto the current release/v3.8.50 tip: wire-capture test passed with 4/4 sampled requests correctly correlated across 11 captured TCP streams — including a real HTTP 400 (deprecated model) and a real HTTP 502 (transient upstream error), both correctly matched between the app-level result and the wire-level status line, not just the happy-200 path. Container + tcpdump fully torn down afterward each run (verified no orphaned podman container or process left running).
  • While testing against a freshly-built image from the current upstream tip, hit and worked around (locally only, not part of this diff) two separate pre-existing base-red issues to actually get a working build/DB: the LLMLingua/onnxruntime native-asset bug already fixed in fix(build): standalone bundle misses LLMLingua dist + onnxruntime native binaries #9712, and a live migration-numbering collision (two files both claiming version 135 — 135_connection_runtime_state.sql and 135_migrate_model_capability_max_token.sql) that will be filed and fixed as its own separate PR next, since it currently breaks any fresh build from release/v3.8.50.

…combo

Temporary WIP commit on this deferred branch — lands in its own separate
PR once the bug-fix extraction batch is done (never bundled into a
bug-fix PR). Unlike liveGeminiShared.ts (provisions its own narrow
2-model Gemini-only combo), this reads the REAL "default" combo
currently configured on the target instance directly from the DB and
exercises every provider/model step in it directly, bypassing combo
routing, so live-test coverage always matches whatever is actually
configured instead of a hardcoded snapshot.

Live-verified against omniroute-beta (seeded with the real 18-model,
5-provider default combo): 14/18 models pass consistently across
non-streaming + streaming Chat Completions and streaming Responses API.
The 4 consistent failures are real external state (cerebras
credits_exhausted, one deprecated openrouter free-tier model), not code
regressions.

(cherry picked from commit c40b13a48fd897259c56f5122e9e57a3dc7654ba)
…test tool

Temporary WIP commit on this deferred branch — lands in the same final
live-test-tool PR as the general default-combo suite, never bundled into
a bug-fix PR.

liveContainerHarness.ts spins up a dedicated, throwaway podman container
(same runner-base image target as the operator's local dev/beta
containers) so wire-capture tests are fully self-contained: builds the
image if missing, starts the container with a persistent data dir, waits
for health, seeds the real "default" combo + provider connections from
the operator's local omniroute-dev instance (idempotent — only runs once
per data dir), and provisions API keys via the running instance's own
auth flow.

wireCapture.ts captures the container's actual network traffic via
`podman unshare nsenter --net=<container netns> -- tcpdump` — no root
needed, verified working live (this generalizes the root-requiring
`sudo nsenter -t $PID` command scripts/sre/tcp-close-analyzer.py already
documented for the same rootless-Podman netns problem; that script's
docstring now documents both). Capture and analysis needed two real fixes
found only by running the pipeline live: `-U` (unbuffered tcpdump writes)
plus a `pkill -f <pcap path>` fallback, since `podman unshare -> nsenter
-> tcpdump` is a 3-level subprocess chain and SIGTERM to the top-level
process doesn't reach the tcpdump grandchild, leaving an orphaned process
and a truncated/unreadable pcap; and filtering on the container's
internal listening port (20128) rather than the dynamically-assigned host
port, since capture happens inside the container's own network namespace
where only the internal port is meaningful.

live-default-combo-wire-capture.test.ts (gated on RUN_LIVE_WIRE_CAPTURE=1)
ties it together: sends a small representative sample of requests through
the real default combo, then cross-checks each one's app-level JSON
status against the actual HTTP status line observed on the wire via
scripts/sre/tcp-close-analyzer.py's stream reassembly — catching bugs
where the app layer claims success but the wire shows a
truncated/reset stream, not just what liveDefaultComboShared.ts's
existing breadth suite already covers.

Live-verified end-to-end: 4/4 sampled requests correlated correctly
across 8 captured TCP streams, container + capture process fully torn
down afterward (verified no orphaned podman container or tcpdump
process left running).

sendModelRequest/filterActiveModelTargets (liveDefaultComboShared.ts) gain
optional baseUrl/apiKey overrides, defaulting to the existing module-level
omniroute-beta target, so the wire-capture suite can point the same
request-sending logic at its own dedicated container instead.

(cherry picked from commit 914a7e42cbe914f257db9f72eedc902ee1532083)
@hartmark
hartmark force-pushed the test/live-default-combo-tool branch from 71b42ca to 818dcf7 Compare August 8, 2026 14:01
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks for the PR. Please address the mandatory items (tests and/or merge blockers) in this branch, then rerun checks before /merge-prs.

2 similar comments
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks for the PR. Please address the mandatory items (tests and/or merge blockers) in this branch, then rerun checks before /merge-prs.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks for the PR. Please address the mandatory items (tests and/or merge blockers) in this branch, then rerun checks before /merge-prs.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Obrigado pelo PR. Mantive a revisão de fix-in-place e não foi possível concluir o ajuste completo aqui:

  • Para os PRs em fork: não consigo aplicar push de correção diretamente na sua branch.
    Por favor, faça um rebase/sync com release/v3.8.50, resolva conflitos se houver, e rode os checks dessa branch.
    Se preferir, posso aplicar a correção na próxima rodada assim que você mandar o branch atualizado ou confirmar que o PR está limpo pra esse merge.

@hartmark

hartmark commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Code landed in #9862

@hartmark hartmark closed this Aug 9, 2026
@hartmark
hartmark deleted the test/live-default-combo-tool branch August 9, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants