docs(OMN-11513): integration testing evidence — Days 1-7 (through 2026-05-29)

[clone45]

Summary

Integration verification evidence for OMN-11513 (monthly integration testing), contractor lane. Read-only verification, document-don't-fix. This branch collects the per-day findings and evidence manifests, culminating in the 2026-05-29 pass.

What is in this branch

• evidence/OMN-11513/integration-test-passes/2026-05-26-day5-findings.md (+ manifest)
• evidence/OMN-11513/integration-test-passes/2026-05-29-bret-day7-findings.md (+ manifest)
• Prior Days 1-7 evidence (earlier commit)

Key results (2026-05-29)

• All six Day 6 fix PRs verified (code and visual): OMN-12366, 12367, 12397, 12399, 12402, 12139. Data source disclosure ("File Mode") honest on every surface tested.
• Delegation regression resolved for the WSL/contractor environment via the sanctioned local registry overlay: 7 of 7 PASS, where prior runs returned all failures.
• Filed OMN-12434 (High, ARCHITECTURAL_RISK): the delegation registry declares model IDs that do not match the served vLLM IDs, so tier 1 returns 404 on every request and the ladder silently escalates. The cheapest tier never serves.
• Dashboard delegation surface audit (file mode), Group A and Group B complete. Candidate findings: 4 DEMO_DEGRADED, 6 COSMETIC (not yet filed; bundling proposed).
• The dashboard cannot demonstrate cost savings in file mode today (three converging surfaces: OMN-12434, empty Savings Proof Pack, empty Cost Comparison widget). This is a file-mode and projection-availability limitation, not a capability gap; live/projection mode is the intended path.
• Trace deep-link (Act 4) does not update a co-resident Trace Explorer (no navigate-to-trace route; mount-only filter). Workaround documented.
• SEA authority verification clean: dashboard_server retired (feat(OMN-8431): add onex_change_control/overseer — migrate 28 overseer wire types from omnibase_compat #157), CLI routed through ONEX (fix(ci): auto-merge template fixes [OMN-8432] #160).

Outstanding

• Task 5 (golden-chain evidence classification) is blocked on access to the omni_home repo (GitHub 404 for this account). Documented separately for Jonah.
• Status: in progress. Task 7 (judge rehearsal) pending.

Severity classifications follow the engagement taxonomy. Findings use observation language.

Summary by CodeRabbit

• Documentation

  • Added comprehensive integration test reports (Days 3–7) with findings, blockers, and demo readiness assessments
  • Added proof-of-life validation documentation with event chains and replay verification
  • Added SEA demo acceptance evidence reports with test coverage summaries

• Chores

  • Updated evidence contracts and added new command receipt records for test verification
  • Added integration test manifests and JSON artifacts tracking test runs, findings severity, and probe outcomes

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR registers and documents the OMN-11513 integration testing milestone evidence, adding contract-gated evidence requirements, an evidence manifest inventory, and comprehensive test findings from Days 1–7 (Day 3 baseline, Day 5 expanded pass report, Day 7 final status) alongside proof-of-life event chains and SEA security acceptance test results.

Changes

OMN-11513 Evidence Milestone

Layer / File(s)	Summary
Evidence contract and receipt registration contracts/OMN-11513.yaml, drift/dod_receipts/OMN-11513/*/command.yaml	New evidence_requirements entry references Days 1–7 bundle in onex_change_control PR #1886; dod_evidence record checks status: PASS and pr_number: 1886; contract_sha256 hashes updated across five existing receipt files to bind evidence verification artifacts.
Evidence manifest and artifact inventory evidence/OMN-11513/EVIDENCE_MANIFEST.md	OMN-11513 manifest documents May integration testing period, engagement details, and inventory of integration-test passes, proof-of-life, and SEA acceptance artifacts by type. Includes resolved vs. open findings tables and label glossary (LOCAL_RESEARCH_ONLY, REPLAY, FIXTURE, LIVE_MODEL_ONLY, REMOTE_READ_ONLY).
Day 3–5 integration milestone baseline report evidence/OMN-11513/integration-test-passes/2026-05-21-first-integration-milestone.md	Day 3 proof-of-life run consolidating SOW-scoped evidence across repos; documents partial pass with four blockers (gemini-2.0-flash HTTP 429 limit, YAML fence wrapping determinism, --agent invoke crash, missing pricing wiring). Day 5 addendum confirms full LIVE 5/5 chain success after one-line fixes; includes evidence pointers and OCC porting guidance.
Day 5 integration pass report (2026-05-24) evidence/OMN-11513/integration-test-passes/2026-05-24-integration-pass-report.md	Comprehensive test across seven workstreams with environment/commit metadata, results table (PASS/DEGRADED/BLOCKED), event-chain correlation IDs, evidence artifacts inventory, remote connectivity checks (runtime/effects/model endpoints), five findings by severity (cosmetic OMN-11972, sandbox failures OMN-11828, resolved model/dashboard fixes OMN-11691/OMN-11429/OMN-11431/OMN-11694, Tailscale ACL advisory), demo readiness (judge command, fallback strategy), and next actions.
Day 5 detailed findings and manifest (2026-05-26) evidence/OMN-11513/integration-test-passes/2026-05-26-day5-findings.md, 2026-05-26-evidence-manifest.json	Baseline capture record with per-widget audit table; projection connectivity findings (stability health vs. unreachable API), dispatch bridge contract/Kafka behavior, command palette compliance (no prohibited POSTs), data source mode matrix, and architectural risks (dead widget registration, fixture disclosure gap). JSON manifest records tested SHAs, branches, data-source modes, screenshot set, API probe results, and severity counts.
Day 7 integration findings and status (2026-05-29) evidence/OMN-11513/integration-test-passes/2026-05-29-bret-day7-findings.md, 2026-05-29-bret-day7-evidence-manifest.json	File-mode-only test status (projection endpoints unavailable); task completion tracking (1–4, 6 complete; 5 blocked on omni_home access; 7 pending). Findings register (architectural risk OMN-12434, demo degraded, cosmetic, observation-only items), blockers (omni_home/OCC access resolved), severity tally, and demo readiness (file mode only safe; expected live failures; workarounds). JSON manifest records in_progress status, tested SHAs/branches, file-mode PR verification, and comprehensive notes on surface verification outcomes.
Proof-of-life event chains and replay evidence evidence/OMN-11513/proof-of-life/2026-05-22-0b2a477-replay-p2-replay/*, 2026-05-24-0b2a477-live-full-chain/golden_event_chain.json, 2026-05-24-62b87c2-live-a1/golden_event_chain.json	Golden event chains (node-generation request/completion) with envelope metadata and sentiment-classifier tool registration payloads (handler hash, sample test inputs/outputs). Replay honesty receipt (P2.F --replay mode) asserts no provider/HTTP activity via grep assertions; confirms fixture-driven invocation validation without LIVE proof path.
SEA demo acceptance and security test evidence (Wave 5) evidence/OMN-11513/sea-demo-acceptance/2026-05-22-wave5-evidence-report.md	Wave 5 security acceptance report: test scope (--agent, --replay, --report exercised; --progressive, --demo, --entropy skipped), P0 baseline, security-negative tests B.1–B.5 with explicit coverage gaps (isolation, display-vs-chain parity incomplete). LIVE vs REPLAY matrix (LIVE failed sandbox rejection of import re; REPLAY succeeded end-to-end). CLI provenance audit shows zero SOW-required labels. Operational observations, demo risks by severity, reproducible issues with code references, and disposition notes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• OmniNode-ai/onex_change_control#980: Directly updates contract_sha256 values in drift/dod_receipts/.../command.yaml receipt files, using the same evidence-binding pattern as this PR's contract hash refreshes.
• OmniNode-ai/onex_change_control#1300: Updates contract_sha256 in drift/dod_receipts/*/command.yaml for hash-bound receipt gating, using the same evidence verification mechanism as this PR (OMN-11424 ticket pattern).
• OmniNode-ai/onex_change_control#1731: Updates drift/dod_receipts/**/command.yaml receipt metadata by changing contract_sha256 values (OMN-12245 release receipts use same DO&D receipt update pattern as OMN-11513).

---

🐰 Evidence hops through the warren fast,
Chain-linking days from first to last,
Each proof a token, each finding blessed,
May's marathon put to the test!
Five receipts signed, the manifest clear,
We've logged the journey milestone here. ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically summarizes the main change: adding integration testing evidence documentation for OMN-11513 covering Days 1-7 through 2026-05-29.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch clone45/omn-11513-integration-evidence-days-1-7

⚔️ Resolve merge conflicts

• Resolve merge conflict in branch clone45/omn-11513-integration-evidence-days-1-7

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}