Skip to content

docs(e2e): keep migration state in issues - #4649

Merged
cv merged 2 commits into
mainfrom
docs/e2e-migration-state-in-issues
Jun 3, 2026
Merged

docs(e2e): keep migration state in issues#4649
cv merged 2 commits into
mainfrom
docs/e2e-migration-state-in-issues

Conversation

@cv

@cv cv commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Move mutable E2E migration tracking out of repository-owned docs and source files so current state lives in the issues and PRs that own the work. This keeps repo docs focused on durable framework guidance and prevents stale per-script status tables from lingering.

Related Issue

Relates to #3588, #4357, and #4378.

Changes

  • Replace test/e2e-scenario/docs/MIGRATION.md with a policy that points current tracking to GitHub issues and PRs.
  • Update test/e2e-scenario/docs/README.md to describe durable framework inputs and avoid repo-local migration status tracking.
  • Remove test/e2e-scenario/scenarios/migration-inventory.ts and the tests that depended on it as a mutable status ledger.
  • Add framework hygiene coverage that rejects reintroducing repo-local migration inventory and stale checklist language.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Documentation

    • Updated E2E scenario documentation structure and migration tracking guidance to reflect new directory layout and external issue-based tracking approach.
  • Tests

    • Refactored E2E migration test suite to focus on repository hygiene, removing dependency on in-repository migration inventory tracking.

@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PR removes in-repository migration inventory tracking and shifts state management to external GitHub issues/PRs. It updates E2E scenario documentation to reflect this policy, removes the migration inventory TypeScript file entirely, and adds test assertions to enforce that migration state is tracked externally and not stored in the repository.

Changes

Migration Inventory Removal and External Tracking Policy

Layer / File(s) Summary
Documentation Policy Shift to External Tracking
test/e2e-scenario/docs/MIGRATION.md, test/e2e-scenario/docs/README.md
MIGRATION.md is rewritten from a detailed migration tracker to a policy document that explicitly excludes in-repo mutable migration state and directs developers to track work via GitHub issues and PRs. README.md is updated to clarify directory layout, describe durable framework inputs, and document that migration evidence is recorded in owning issues/PRs rather than in-repo coverage reports.
Migration Tracking Hygiene Test
test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts
The test suite is renamed and rewritten from "hybrid scenario migration inventory lock" to "E2E migration tracking hygiene." It now reads MIGRATION.md and README.md and enforces three repository hygiene rules: absence of the migration inventory TypeScript file, presence of documentation text directing tracking to GitHub issues/PRs, and absence of stale per-script migration checklist patterns.
Inventory Dependency Cleanup
test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts
The migrationInventory import is removed and the scenarioOwnerIds() helper test that verified canonical scenario IDs are registered is deleted. The scenario registry test suite now begins with the duplicate-scenario-id test and continues with CLI/registry behavior checks.

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 Out with the ledgers tucked in the trees,
In with the issues that flow on the breeze—
GitHub's the stage, PRs show the way,
Tests guard the repo, no stale state to stay! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: moving mutable migration tracking state from repository files into GitHub issues, which is the core intent of the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/e2e-migration-state-in-issues

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No E2E jobs are recommended. This PR only updates E2E scenario documentation, framework-test assertions, and removes a migration tracking inventory that is no longer imported by runtime scenario execution. It does not change product runtime code, installer/onboarding behavior, sandbox lifecycle, credentials, security boundaries, network policy, inference routing, deployment, or real assistant user flows.

Optional E2E

  • None.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: None
Optional scenario E2E: None

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

  • None. Changes are limited to scenario documentation, framework/unit tests, and removal of the non-runtime migration inventory. The deleted migration-inventory.ts is not imported by scenario runtime, workflows, scenario metadata, expected-state metadata, or suite metadata, so it should not affect live scenario E2E behavior.

Optional scenario E2E

  • None.

Relevant changed files

  • test/e2e-scenario/docs/MIGRATION.md
  • test/e2e-scenario/docs/README.md
  • test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts
  • test/e2e-scenario/scenarios/migration-inventory.ts

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 1 still applies, 0 new items found

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • Source-of-truth review needed: E2E migration tracking policy in MIGRATION.md, README.md, and the deleted migration inventory: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: This follow-up is covered by the security finding: external tracking must remain auditable for security-sensitive migration coverage because the deleted inventory previously served as a repo-local audit ledger.
  • Confirm security-sensitive migration coverage remains auditable after deleting the inventory lock (test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts:19): The PR deletes the machine-readable migration inventory and replaces inventory coverage tests with hygiene tests that enforce external tracking. This does not create a direct sandbox, credential, SSRF, policy, workflow, or auth vulnerability, but it moves migration audit state for security-sensitive E2E suites out of the repository. The deleted ledger previously mapped expected states, validation suites, validation-suite scripts, and assertion groups, including credentials, policy, injection, sandbox lifecycle, and messaging no-secret-leak coverage.
    • Recommendation: Before relying on the new policy long term, ensure the referenced issues and PRs preserve an auditable mapping for security-sensitive migration coverage, or keep a minimal non-status in-repo invariant that verifies those security domains remain represented without reintroducing mutable per-script status.
    • Evidence: The diff deletes test/e2e-scenario/scenarios/migration-inventory.ts and removes inventory-based tests. Nearby tests still mitigate much of the risk by verifying suite-to-assertion-group mapping, script existence, assertion-group usage, coverage gaps, and scenario suite hygiene; however, no replacement in-repo invariant specifically audits the external security-sensitive migration mapping.

🌱 Nice ideas

  • None.
Since last review details

Current findings:

  • Source-of-truth review needed: E2E migration tracking policy in MIGRATION.md, README.md, and the deleted migration inventory: The advisor marked localized patch analysis as needs_followup.
    • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
    • Evidence: This follow-up is covered by the security finding: external tracking must remain auditable for security-sensitive migration coverage because the deleted inventory previously served as a repo-local audit ledger.
  • Confirm security-sensitive migration coverage remains auditable after deleting the inventory lock (test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts:19): The PR deletes the machine-readable migration inventory and replaces inventory coverage tests with hygiene tests that enforce external tracking. This does not create a direct sandbox, credential, SSRF, policy, workflow, or auth vulnerability, but it moves migration audit state for security-sensitive E2E suites out of the repository. The deleted ledger previously mapped expected states, validation suites, validation-suite scripts, and assertion groups, including credentials, policy, injection, sandbox lifecycle, and messaging no-secret-leak coverage.
    • Recommendation: Before relying on the new policy long term, ensure the referenced issues and PRs preserve an auditable mapping for security-sensitive migration coverage, or keep a minimal non-status in-repo invariant that verifies those security domains remain represented without reintroducing mutable per-script status.
    • Evidence: The diff deletes test/e2e-scenario/scenarios/migration-inventory.ts and removes inventory-based tests. Nearby tests still mitigate much of the risk by verifying suite-to-assertion-group mapping, script existence, assertion-group usage, coverage gaps, and scenario suite hygiene; however, no replacement in-repo invariant specifically audits the external security-sensitive migration mapping.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts`:
- Around line 23-30: The test "documents that migration state lives in issues
and PRs" currently only asserts README contains a generic phrase; update it to
explicitly assert that README.md still mentions "GitHub issues and pull
requests" (or the exact guidance string used in MIGRATION_DOC) so it fails if
the README loses the issues/PRs guidance — locate the test (it(...) in
e2e-migration-inventory-lock.test.ts), use the same read(README_DOC) call and
add an expect(readme).toContain("GitHub issues and pull requests") (or match the
exact wording used in MIGRATION_DOC) to tighten the assertion.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ff33fc6d-cf35-4dd5-8880-7dd26b2b4fd0

📥 Commits

Reviewing files that changed from the base of the PR and between 798d5a3 and c7a6377.

📒 Files selected for processing (5)
  • test/e2e-scenario/docs/MIGRATION.md
  • test/e2e-scenario/docs/README.md
  • test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts
  • test/e2e-scenario/scenarios/migration-inventory.ts
💤 Files with no reviewable changes (2)
  • test/e2e-scenario/scenarios/migration-inventory.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts

Comment on lines +23 to 30
it("documents that migration state lives in issues and PRs", () => {
const migration = read(MIGRATION_DOC);
const readme = read(README_DOC);

expect(runSource).not.toContain("migration-inventory");
expect(migration).toMatch(/tracked\s+outside the repository/);
expect(migration).toContain("GitHub issues and pull requests");
expect(readme).toMatch(/Migration status is tracked outside the repository/);
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Assert the README still points to issues/PRs.

This test only checks that README.md says migration status is tracked "outside the repository". It would still pass if the README lost the issue/PR guidance that this PR is trying to preserve.

Suggested tightening
   it("documents that migration state lives in issues and PRs", () => {
     const migration = read(MIGRATION_DOC);
     const readme = read(README_DOC);

     expect(migration).toMatch(/tracked\s+outside the repository/);
     expect(migration).toContain("GitHub issues and pull requests");
     expect(readme).toMatch(/Migration status is tracked outside the repository/);
+    expect(readme).toMatch(/parent epic/i);
+    expect(readme).toMatch(/pull requests?/i);
   });
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
it("documents that migration state lives in issues and PRs", () => {
const migration = read(MIGRATION_DOC);
const readme = read(README_DOC);
expect(runSource).not.toContain("migration-inventory");
expect(migration).toMatch(/tracked\s+outside the repository/);
expect(migration).toContain("GitHub issues and pull requests");
expect(readme).toMatch(/Migration status is tracked outside the repository/);
});
it("documents that migration state lives in issues and PRs", () => {
const migration = read(MIGRATION_DOC);
const readme = read(README_DOC);
expect(migration).toMatch(/tracked\s+outside the repository/);
expect(migration).toContain("GitHub issues and pull requests");
expect(readme).toMatch(/Migration status is tracked outside the repository/);
expect(readme).toMatch(/parent epic/i);
expect(readme).toMatch(/pull requests?/i);
});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-scenario/framework-tests/e2e-migration-inventory-lock.test.ts`
around lines 23 - 30, The test "documents that migration state lives in issues
and PRs" currently only asserts README contains a generic phrase; update it to
explicitly assert that README.md still mentions "GitHub issues and pull
requests" (or the exact guidance string used in MIGRATION_DOC) so it fails if
the README loses the issues/PRs guidance — locate the test (it(...) in
e2e-migration-inventory-lock.test.ts), use the same read(README_DOC) call and
add an expect(readme).toContain("GitHub issues and pull requests") (or match the
exact wording used in MIGRATION_DOC) to tighten the assertion.

@cv
cv requested review from cjagwani and prekshivyas June 3, 2026 16:35

@prekshivyas prekshivyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Direction is right and the diff is clean.

Non-blocking nit: the hygiene test in e2e-migration-inventory-lock.test.ts only checks the "tracked outside the repository" string. Worth tightening to assert MIGRATION.md still references #3588 (parent epic) and README.md still points to it, so the policy's actual intent — pointing readers somewhere — is what's guarded. CodeRabbit's line-30 nit captures the same idea. Happy if you address it here or as a follow-up.

@cv
cv merged commit 4a7a04a into main Jun 3, 2026
28 checks passed
@cv
cv deleted the docs/e2e-migration-state-in-issues branch June 3, 2026 17:19
@wscurran wscurran added the chore Build, CI, dependency, or tooling maintenance label Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants