Skip to content

fix(ci): eliminate dev test failure baseline - #151

Merged
bloodf merged 1 commit into
devfrom
fix/dev-zero-tests
Jul 10, 2026
Merged

bloodf merged 1 commit into
devfrom
fix/dev-zero-tests

Conversation

@bloodf

@bloodf bloodf commented Jul 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

This is Stage 2 of the dev-recovery program. It starts from the verified
post-#124 dev, fixes the remaining deterministic regressions, deletes all 51
stale failure-baseline rows, and restores a zero-raw-failure contract before
the normal PR queue resumes.

  • Base: dev@6457d95dd5e16e25cdd758d32dad904d16390256
  • Head: 9d73875704a2fec0683ea514a2bb40c7a7c06395
  • Baseline: 51 entries -> 0 entries; no additions
  • Final fail-closed result: raw 0 / known 0 / stale 0

No feature or port PR should merge until this PR is squash-merged and the
post-merge dev checks are green. Nightly remains manually disabled until
that post-merge verification completes.

Authoritative starting failures

GitHub Actions run 29073976751 / artifact 8220013321 on the exact base
reported 17 raw failures, all 17 known, 34 stale rows, and zero additions:

  1. Kiro full-body golden request snapshot.
  2. AgentRouter Claude passthrough/context contract.
  3. AgentRouter's invalid mixed multi-transport assertion.
  4. No-auth catalogs hidden by unrelated connections.
  5. Command Code validation model/route selection.
  6. GitLab Duo environment base-URL fallback.
  7. AgentRouter alias registration.
  8. AgentRouter local icon registration.
  9. Api.airforce validation source-regex assertion.
  10. Suggested-model OpenAI filtering.
  11. Static registry export coverage.
  12. Copilot/M365 specialized executor registration.
  13. VeoAI Free video executor routing.
  14. Pollinations missing-connection public fallback.
  15. Pollinations excluded-connection public fallback.
  16. Pollinations keyless validation.
  17. ZenMux Free specialized executor registration.

Runtime and CI changes

  • Preserve dotted Kiro catalog IDs on the wire and remove only synthetic
    thinking/agentic suffixes.
  • Restore AgentRouter's single Claude /v1/messages transport, aliases, icon,
    passthrough behavior, and context contract.
  • Give both Command Code IDs an explicit validation model and share its probe
    status policy between validation and saved-connection health checks.
  • Restore GitLab Duo base-URL precedence, including the legacy environment
    variable.
  • Keep no-auth catalogs visible; implement Pollinations dual-auth behavior,
    prefer real keys, never forward placeholders, and hide premium-only models
    without a usable key.
  • Use a negative explicit non-chat filter for OpenAI-compatible model catalogs.
    The fetch route accepts only exact registry-declared URL/filter pairs and
    refuses redirects.
  • Register Copilot Web, M365 Copilot, VeoAI Free, ZenMux Free, and supported
    aliases with their specialized executors; regenerate AGENT-INDEX.md.
  • Harden Copilot/M365/ZenMux session handling: bounded asynchronous hashcash,
    no token-keyed session pool, strict M365 host/path validation, credential-safe
    diagnostics, and abort propagation for streaming and non-streaming readers.
  • Redact credential headers, JSON/body fields, query/userinfo/fragment values,
    response data, stream chunks, and error messages/stacks in request logging.
  • Make Nightly build state isolated and keep publication explicitly gated on
    all successful install/lint/index/build/test steps.

Source provenance and intentional divergences

  • OmniRoute source refreshed at
    9cd18bf9a11b7d2e8c037c374631b492adabf469.
  • Upstream 9router source refreshed at
    b10b8070632cbdeb3c6b26b0ccf60f433ebf5014.
  • Dotted Kiro IDs follow live-verified OmniRoute PR #6170 merge b3a2cfe;
    conflicting dash-conversion experiments were rejected.
  • PR test(baseline): drain remaining 3-day window regressions (batch 2) #126 was reviewed only as a patch bank at
    d5acb2fa909e4cf0419ef7cef528db312e5ae2d8:
    • retained/improved: no-auth visibility and ZenMux registration;
    • rewritten: AgentRouter assertions, suggested-model filtering,
      Pollinations fallback, and web-session routing;
    • already landed: Api.airforce canonical validateUrl probing;
    • rejected: baseline additions, positive model allowlisting, unused-helper
      edits, generated-index naming workarounds, and mixed AgentRouter transport.

No #126 commit or baseline hunk was cherry-picked.

Verification

All commands used disposable HOME and DATA_DIR and Node 20.20.2 / npm
10.8.2.

Clean Linux pass 1 (fresh container/cache, root + tests npm ci): exit 0
  280 files passed, 12 skipped; 2,556 passed, 18 expected-fail, 59 skipped
  2,633 assertions total
  log sha256 acc4a0639b7b613cc928699de392abba05cdbedbe6b87d1d513aa46e29ec9281

Clean Linux pass 2 (independent fresh container/cache): exit 0
  identical counts
  log sha256 b06f8fecdb8a82719052aceaa5719b716502a4fe22a5d47ddfcf72124dd753aa

Final fail-closed gate: exit 0
  292 suites / 2,633 assertions / failures 0 / runtime errors 0
  raw 0 / known 0 / stale 0 / baseline additions none
  JSON  sha256 9380115571a3e320039a69592ef800c55de6c417711063cd3e412e3b51662422
  JUnit sha256 03fc8ede15d12b5308e0f9367575c0870685e2bcc5087a5faac6048315416cdd

Focused QA: 25 files / 548 tests passed
npm run lint: exit 0 (181 pre-existing warnings, 0 errors)
npm run check:agent-index: exit 0
npm run build: exit 0; all DB paths under the temporary DATA_DIR
commitlint origin/dev..HEAD: exit 0
git diff --check: clean

The full suite still emits non-failing MaxListenersExceededWarning diagnostics
for DB shutdown listeners. This is pre-existing diagnostic debt, not a hidden
test failure; JSON/JUnit report no unhandled or runtime errors.

GitHub Ubuntu x64 run 29077381388 passed twice on fresh runners. Attempt 1
artifact 8221325378 and attempt 2 artifact 8221414954 each contain 292
suites / 2,633 assertions, zero failures/runtime-error suites, JUnit failures
0, raw/known/stale 0, and no baseline additions. Lint/build run 29077381460
and commitlint run 29077381416 also passed on both attempts.

Provider-authorized live smoke tests were not run because this change must not
read operator credentials.

Documentation, migration, and compatibility

  • Added docs/development/dev-zero-test-recovery.md and updated recovery,
    provider-port, test, and Nightly documentation.
  • Added focused tests for every behavior change, SSRF controls, validation,
    cancellation, and credential redaction.
  • No database migration is added.
  • Stored API-key secrets are not rewritten, rotated, exposed, or re-keyed.
  • Legacy commandcode, GitLab environment, provider-name, header, API-key,
    and data-path wire compatibility is preserved.

Merge checklist

  • Empty known-fails.txt; no additions
  • Two independent clean Linux installs and direct raw-zero runs
  • Final fail-closed JSON/JUnit reports parse and contain zero failures
  • Lint, generated index, isolated build, commitlint, and diff hygiene pass
  • Staff review: no local blocker
  • QA review: no local blocker
  • Required GitHub checks green twice on fresh Ubuntu runners
  • All review threads resolved (zero threads at final pre-merge refresh)
  • Squash-merged as d0663fd60d782a74aa85bbdd5d0959ad41027e68
  • Post-merge dev runs 29077864260 and 29077864283 green; artifact
    8221524665 reports raw/known/stale 0 and 2,633 assertions
  • Nightly re-enabled; test(baseline): drain remaining 3-day window regressions (batch 2) #126 closed with the patch-bank audit

@bloodf
bloodf merged commit d0663fd into dev Jul 10, 2026
6 checks passed
@bloodf
bloodf deleted the fix/dev-zero-tests branch July 17, 2026 02:49
@bloodf bloodf mentioned this pull request Jul 20, 2026
7 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant