Repository navigation
test(integration): add general live-test tool for the real "default" combo + rootless wire capture - #9744
Closed
hartmark wants to merge 2 commits into
Closed
test(integration): add general live-test tool for the real "default" combo + rootless wire capture#9744hartmark wants to merge 2 commits into
hartmark wants to merge 2 commits into
Conversation
hartmark
force-pushed
the
test/live-default-combo-tool
branch
from
August 8, 2026 11:46
8091edd to
71b42ca
Compare
…combo Temporary WIP commit on this deferred branch — lands in its own separate PR once the bug-fix extraction batch is done (never bundled into a bug-fix PR). Unlike liveGeminiShared.ts (provisions its own narrow 2-model Gemini-only combo), this reads the REAL "default" combo currently configured on the target instance directly from the DB and exercises every provider/model step in it directly, bypassing combo routing, so live-test coverage always matches whatever is actually configured instead of a hardcoded snapshot. Live-verified against omniroute-beta (seeded with the real 18-model, 5-provider default combo): 14/18 models pass consistently across non-streaming + streaming Chat Completions and streaming Responses API. The 4 consistent failures are real external state (cerebras credits_exhausted, one deprecated openrouter free-tier model), not code regressions. (cherry picked from commit c40b13a48fd897259c56f5122e9e57a3dc7654ba)
…test tool Temporary WIP commit on this deferred branch — lands in the same final live-test-tool PR as the general default-combo suite, never bundled into a bug-fix PR. liveContainerHarness.ts spins up a dedicated, throwaway podman container (same runner-base image target as the operator's local dev/beta containers) so wire-capture tests are fully self-contained: builds the image if missing, starts the container with a persistent data dir, waits for health, seeds the real "default" combo + provider connections from the operator's local omniroute-dev instance (idempotent — only runs once per data dir), and provisions API keys via the running instance's own auth flow. wireCapture.ts captures the container's actual network traffic via `podman unshare nsenter --net=<container netns> -- tcpdump` — no root needed, verified working live (this generalizes the root-requiring `sudo nsenter -t $PID` command scripts/sre/tcp-close-analyzer.py already documented for the same rootless-Podman netns problem; that script's docstring now documents both). Capture and analysis needed two real fixes found only by running the pipeline live: `-U` (unbuffered tcpdump writes) plus a `pkill -f <pcap path>` fallback, since `podman unshare -> nsenter -> tcpdump` is a 3-level subprocess chain and SIGTERM to the top-level process doesn't reach the tcpdump grandchild, leaving an orphaned process and a truncated/unreadable pcap; and filtering on the container's internal listening port (20128) rather than the dynamically-assigned host port, since capture happens inside the container's own network namespace where only the internal port is meaningful. live-default-combo-wire-capture.test.ts (gated on RUN_LIVE_WIRE_CAPTURE=1) ties it together: sends a small representative sample of requests through the real default combo, then cross-checks each one's app-level JSON status against the actual HTTP status line observed on the wire via scripts/sre/tcp-close-analyzer.py's stream reassembly — catching bugs where the app layer claims success but the wire shows a truncated/reset stream, not just what liveDefaultComboShared.ts's existing breadth suite already covers. Live-verified end-to-end: 4/4 sampled requests correlated correctly across 8 captured TCP streams, container + capture process fully torn down afterward (verified no orphaned podman container or tcpdump process left running). sendModelRequest/filterActiveModelTargets (liveDefaultComboShared.ts) gain optional baseUrl/apiKey overrides, defaulting to the existing module-level omniroute-beta target, so the wire-capture suite can point the same request-sending logic at its own dedicated container instead. (cherry picked from commit 914a7e42cbe914f257db9f72eedc902ee1532083)
hartmark
force-pushed
the
test/live-default-combo-tool
branch
from
August 8, 2026 14:01
71b42ca to
818dcf7
Compare
Collaborator
|
Thanks for the PR. Please address the mandatory items (tests and/or merge blockers) in this branch, then rerun checks before /merge-prs. |
2 similar comments
Collaborator
|
Thanks for the PR. Please address the mandatory items (tests and/or merge blockers) in this branch, then rerun checks before /merge-prs. |
Collaborator
|
Thanks for the PR. Please address the mandatory items (tests and/or merge blockers) in this branch, then rerun checks before /merge-prs. |
Collaborator
|
Obrigado pelo PR. Mantive a revisão de
|
Contributor
Author
|
Code landed in #9862 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two pieces of live-test infrastructure, gated behind explicit env vars so they never run in CI (need podman, real provider credentials, and — for the wire-capture piece — tcpdump/python3):
1. General default-combo live-test tool (
RUN_LIVE_WIRE_CAPTURE=1is for piece 2 below; this piece reusesomniroute-beta-style gating viaOMNIROUTE_API_KEY) —tests/integration/liveDefaultComboShared.ts+live-default-combo-workload.test.ts. Unlike the existingliveGeminiShared.ts(which provisions its own narrow 2-model Gemini-only combo), this reads the REALdefaultcombo currently configured on the target instance directly from the DB and exercises every provider/model step in it directly, bypassing combo routing — so live-test coverage always matches whatever the operator actually has configured instead of a hardcoded snapshot that goes stale the moment the combo changes. Live-verified earlier this session: 14/18 models pass consistently across non-streaming + streaming Chat Completions and streaming Responses API against the real 18-model/5-providerdefaultcombo; the 4 consistent failures were real external state (a provider temporarily out of credits, one deprecated free-tier model), not code regressions.2. Rootless wire-capture correlation (
RUN_LIVE_WIRE_CAPTURE=1) —tests/integration/liveContainerHarness.ts,wireCapture.ts,live-default-combo-wire-capture.test.ts. Spins up a dedicated, throwaway podman container running the target branch's own code, captures its real wire traffic viapodman unshare nsenter --net=<container netns> -- tcpdump(no root needed — verified working live), sends a sample of real requests through the realdefaultcombo, and cross-checks each one's app-level JSON result against the actual HTTP status line + TCP behavior observed on the wire viascripts/sre/tcp-close-analyzer.py's stream reassembly (that script's own docstring now documents the rootless capture command alongside the existing root-requiring one). Catches bugs where the app layer claims one thing but the wire shows another — e.g. a truncated/reset stream, or a status mismatch.Building this surfaced two real, non-obvious bugs along the way (both fixed inline, not part of this PR's own diff since they're general build-tooling issues found via
liveContainerHarness.tsneeding to actually build the target image):podman unshare -> nsenter -> tcpdumpis a 3-level subprocess chain;SIGTERMto the top-level tracked process doesn't reliably reach the tcpdump grandchild, leaving orphaned processes and a truncated/unreadable pcap. Fixed with-U(unbuffered tcpdump writes) + apkill -f <pcap path>fallback inwireCapture.ts's own cleanup.Test plan
npm run typecheck:core— cleannpm run lint— cleannpm run check:file-size— cleanrelease/v3.8.50tip: wire-capture test passed with 4/4 sampled requests correctly correlated across 11 captured TCP streams — including a realHTTP 400(deprecated model) and a realHTTP 502(transient upstream error), both correctly matched between the app-level result and the wire-level status line, not just the happy-200 path. Container + tcpdump fully torn down afterward each run (verified no orphaned podman container or process left running).135—135_connection_runtime_state.sqland135_migrate_model_capability_max_token.sql) that will be filed and fixed as its own separate PR next, since it currently breaks any fresh build fromrelease/v3.8.50.