Skip to content

test(testing): demonstrate missing regression evidence command (sc-2408) #537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
norvalbv wants to merge 1 commit into from
Closed

test(testing): demonstrate missing regression evidence command (sc-2408) #537

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions cli/__tests__/help-cli.test.mts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@
'reconcile',
'guard-branch',
'base-status',
'prove-regression',
]) {
expect(r.stdout).toContain(`devkit ${name}`);

Check failure on line 24 in cli/__tests__/help-cli.test.mts

View workflow job for this annotation

GitHub Actions / gate

[parallel] cli/__tests__/help-cli.test.mts > devkit help surface > `--help` lists every command (derived from meta) 

AssertionError: expected 'devkit — wire a repo onto @norvalbv/d…' to contain 'devkit prove-regression'

- Expected
+ Received

- devkit prove-regression
+ devkit — wire a repo onto @norvalbv/devkit's shared configs + gate-engine.
+
+ Usage:
+   devkit <command> [options]
+
+ Commands:
+   devkit init          Wire this repo onto devkit (interactive wizard; idempotent).
+   devkit doctor        Diagnose drift for the installed component set (read-only).
+   devkit clean         Uninstall devkit — reverse init for the recorded mode.
+   devkit sync-skills   Copy bundled skills into selected agent providers.
+   devkit sync-agents   Copy review/testing agents into selected agent providers.
+   devkit sync-hooks    Copy devkit agent-hook scripts into selected providers (registration-free).
+   devkit sync-hook-runner Make this checkout run its OWN hooks (stage the runner; replace a sibling hooksPath pin).
+   devkit release       MAINTAINER-ONLY: bump version, test, and open a release PR.
+   devkit update        Self-update devkit to the latest published tag.
+   devkit upgrade       Fully reconcile this repo to the installed devkit (one idempotent pass).
+   devkit move          Relocate source files + rewrite every reference.
+   devkit oxc           Run Devkit's pinned native Oxlint or Oxfmt (no consumer install).
+   devkit anti-slop     Check vendored anti-slop rules with an explicit shrink-only baseline.
+   devkit reconcile     After a PR merges, refresh stale files in a shared checkout.
+   devkit ship          Commit files onto a new branch + open a PR without moving HEAD.
+   devkit review        Review a trusted checkout without committing or changing it.
+   devkit guard-branch  PreToolUse hook: block direct commits on a protected branch.
+   devkit base-status   Report whether origin/<base> moved under this checkout, and which paths it touched.
+   devkit coverage-run  Run vitest with coverage in an isolated reports dir (parallel-agent safe).
+   devkit test-report-run Run vitest and emit a machine-readable per-file test report for CI.
+   devkit baseline-status Report which test files are failing on the default branch (and whether yours is).
+   devkit --version    Print devkit's version.
+   devkit --help       This help.
+
+ Docs: see docs/glossary.md + docs/troubleshooting.md in the devkit repo.
+ Run `devkit help <command>` (or `devkit <command> --help`) for the full options of any command.
+

 ❯ cli/__tests__/help-cli.test.mts:24:24
}
});

Expand Down Expand Up @@ -66,6 +67,37 @@
expect(r.stdout).toMatch(/devkit reconcile —/);
});

it('documents the generic captured-evidence contract', () => {
const r = run(['prove-regression', '--help']);
expect(r.status).toBe(0);

Check failure on line 72 in cli/__tests__/help-cli.test.mts

View workflow job for this annotation

GitHub Actions / gate

[parallel] cli/__tests__/help-cli.test.mts > devkit help surface > documents the generic captured-evidence contract 

AssertionError: expected 1 to be +0 // Object.is equality

- Expected
+ Received

- 0
+ 1

 ❯ cli/__tests__/help-cli.test.mts:72:22
expect(r.stdout).toContain('--red <ref>');
expect(r.stdout).toContain('-- <test command>');
expect(r.stdout).toContain('--vitest-report');
expect(r.stdout).toMatch(/CAPTURED execution\s+evidence, not\s+automatic proof/i);
expect(r.stdout).toMatch(/sampled boundary\s+fingerprints matched/i);
expect(r.stdout).toMatch(/not an atomic filesystem snapshot/i);
expect(r.stdout).toMatch(/each\s+operand receives an independent copy/i);
expect(r.stdout).not.toMatch(/node_modules is linked|mutable exception/i);
expect(r.stdout).toMatch(/Windows interruption terminates\s+only the direct helper/i);
});

it('leaves --help after prove-regression command boundary in the child argv', () => {
const r = run([
'prove-regression',
'--red',
'definitely-not-a-ref',
'--green',
'definitely-not-a-ref',
'--',
'node',
'test.mjs',
'--help',
]);
expect(r.status).toBe(1);
expect(r.stdout).not.toMatch(/devkit prove-regression —/);
expect(r.stderr).toContain('prove-regression setup:');

Check failure on line 98 in cli/__tests__/help-cli.test.mts

View workflow job for this annotation

GitHub Actions / gate

[parallel] cli/__tests__/help-cli.test.mts > devkit help surface > leaves --help after prove-regression command boundary in the child argv 

AssertionError: expected 'devkit: unknown command "prove-regres…' to contain 'prove-regression setup:'

- Expected
+ Received

- prove-regression setup:
+ devkit: unknown command "prove-regression"
+
+

 ❯ cli/__tests__/help-cli.test.mts:98:22
});

it('documents the review trust boundary and target/base options', () => {
const r = run(['review', '--help']);
expect(r.status).toBe(0);
Expand Down
Loading