Skip to content

chore(quality): expand check:release-green to the full release-PR gate set #4758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
diegosouzapw merged 1 commit into from
Jun 23, 2026
Merged

chore(quality): expand check:release-green to the full release-PR gate set #4758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 49 additions & 8 deletions scripts/quality/validate-release-green.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,32 @@
// the real state of the release branch at any time.
//
// DESIGN — never blocking to contributors:
// • HARD checks (typecheck, lint errors, unit, vitest, db-rules, public-creds,
// optionally package-artifact) → a failure here is a real defect; exit 1.
// • DRIFT checks (eslint WARNINGS, cognitive-complexity, file-size) → ratchet
// drift accrued across the cycle is NOT a contributor's fault; it is reported
// and rebaselined by the maintainer at release. Drift NEVER changes the exit
// code, so wiring this as a check can never block anyone on drift.
// • HARD checks (typecheck, lint errors, db-rules, public-creds, docs-all,
// unit, vitest, integration, optionally package-artifact) → a failure here is
// a real defect; exit 1.
// • DRIFT checks (eslint WARNINGS, cognitive-complexity, file-size, cyclomatic
// complexity, dead-code, type-coverage, compression-budget, openapi-coverage,
// workflow-lint/zizmor, codeql-ratchet) → ratchet drift accrued across the
// cycle is NOT a contributor's fault; it is reported and rebaselined by the
// maintainer at release. Drift NEVER changes the exit code, so wiring this as
// a check can never block anyone on drift.
//
// COMPLETENESS: this mirrors the FULL release-PR gate set (quality-gate +
// quality-extended + docs-sync-strict + integration), not a subset — and reports
// EVERY red in one pass (the report is collected, not fail-fast), so the release
// PR is green on its first CI run instead of revealing reds in ~40-min layers. The
// only release-PR gates it cannot reproduce locally are GitHub-side CodeQL semantic
// analysis and SonarQube/SonarCloud (external services).
//
// This script DIAGNOSES + REPORTS only (no auto-fix). The fix-to-green
// orchestration lives in the (future) /green-prs + review-prs flows that call it.
// orchestration lives in the /green-prs + review-prs flows that call it.
//
// Usage:
// node scripts/quality/validate-release-green.mjs [--json] [--with-build] [--quick]
// --json emit machine-readable JSON to stdout (report goes to stderr)
// --with-build also run check:pack-artifact (needs a dist/ build — slow)
// --quick skip the slow unit + vitest suites (drift + typecheck + lint only)
// --quick skip the slow unit + vitest + integration suites (drift + fast
// gates only)

import { execFileSync } from "node:child_process";
import { readFileSync } from "node:fs";
Expand Down Expand Up @@ -144,6 +155,16 @@ function main() {
record({ id, label, kind: "hard", ok: code === 0, detail: code === 0 ? "pass" : firstFailureLine(out) });
};

// A ratchet command (check:complexity, check:dead-code, …) exits 1 ONLY on a
// measured regression and self-skips (exit 0) when its tooling is absent — so a
// non-zero exit here is drift to rebaseline at release, never a contributor block.
// ALL checks run regardless of earlier failures (the report is collected, not
// fail-fast) so one pass surfaces every red instead of revealing them in layers.
const driftCmd = (id, label, cmd, cmdArgs, okDetail = "within baseline") => {
const { code, out } = run(cmd, cmdArgs);
record({ id, label, kind: "drift", ok: code === 0, detail: code === 0 ? okDetail : firstFailureLine(out) });
};
Comment on lines +158 to +166

@gemini-code-assist gemini-code-assist Bot Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Issue: Critical Configuration/Tooling Errors Treated as Non-Blocking Drift

Currently, driftCmd treats any non-zero exit code as a non-blocking drift. However, the underlying quality scripts (such as check-complexity.mjs, check-dead-code.mjs, and check-type-coverage.mjs) explicitly distinguish between:

  1. Exit code 1: A measured regression (which is indeed drift and should be non-blocking).
  2. Exit code 2 (or other non-zero codes): Critical setup, configuration, or tooling errors (e.g., a missing baseline file or a tool execution crash).

By treating all non-zero exit codes as drift, a critical configuration error (like a missing baseline file) will silently be ignored and will not block the release PR.

Recommendation

Update driftCmd to classify exit codes greater than 1 as hard failures so that critical configuration or execution errors correctly block the release.

  // A ratchet command (check:complexity, check:dead-code, …) exits 1 ONLY on a
  // measured regression and self-skips (exit 0) when its tooling is absent — so a
  // non-zero exit here is drift to rebaseline at release, never a contributor block.
  // ALL checks run regardless of earlier failures (the report is collected, not
  // fail-fast) so one pass surfaces every red instead of revealing them in layers.
  const driftCmd = (id, label, cmd, cmdArgs, okDetail = "within baseline") => {
    const { code, out } = run(cmd, cmdArgs);
    const isCriticalError = code > 1;
    record({
      id,
      label,
      kind: isCriticalError ? "hard" : "drift",
      ok: code === 0,
      detail: code === 0 ? okDetail : firstFailureLine(out),
    });
  };


process.stderr.write("🔎 Release-green validation (current working tree)\n\n");

hardCmd("typecheck", "Typecheck (core)", npmCmd, ["run", "typecheck:core"]);
Expand Down Expand Up @@ -205,9 +226,29 @@ function main() {
});
}

// Remaining quality-gate / quality-extended ratchets that the PR→release
// fast-gates skip and that historically surfaced — one at a time, because the
// CI Quality Ratchet job is fail-fast — only on the release PR. Running them all
// here (drift, never blocking) means a single rebaseline pass at release.
driftCmd("complexity", "Cyclomatic complexity (ratchet)", npmCmd, ["run", "check:complexity"]);
driftCmd("dead-code", "Dead-code (ratchet)", npmCmd, ["run", "check:dead-code"]);
driftCmd("type-coverage", "Type coverage (ratchet)", npmCmd, ["run", "check:type-coverage"]);
driftCmd("compression-budget", "Compression budget (ratchet)", npmCmd, ["run", "check:compression-budget"]);
driftCmd("openapi-coverage", "OpenAPI route coverage (ratchet)", npmCmd, ["run", "check:openapi-coverage"]);
driftCmd("workflow-lint", "Workflow lint (zizmor ratchet)", npmCmd, ["run", "check:workflows", "--", "--ratchet"]);
driftCmd("codeql-ratchet", "CodeQL alerts (ratchet)", npmCmd, ["run", "check:codeql-ratchet"]);

// Docs sync + fabricated-docs (strict) is a real-defect gate (invented env vars /
// routes, i18n mirror drift) — HARD.
hardCmd("docs-all", "Docs sync + fabricated-docs (strict)", npmCmd, ["run", "check:docs-all"]);

if (!QUICK) {
hardCmd("unit", "Unit tests (full, CI concurrency)", npmCmd, ["run", "test:unit:ci"]);
hardCmd("vitest", "Vitest (MCP / autoCombo / cache)", npmCmd, ["run", "test:vitest"]);
// Integration tests run ONLY on the release PR full CI (PR→main), so an assertion
// regression here (e.g. a contributor flipping a Codex fingerprint key order) is
// invisible until release — run them in the pre-flight as a HARD gate.
hardCmd("integration", "Integration tests", npmCmd, ["run", "test:integration"]);
}
if (WITH_BUILD) {
hardCmd("pack-artifact", "Package artifact (npm pack policy)", npmCmd, ["run", "check:pack-artifact"]);
Expand Down
24 changes: 24 additions & 0 deletions tests/unit/validate-release-green.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,27 @@ test("computeVerdict: releaseGreen iff zero HARD failures (drift never blocks)",
]);
assert.equal(allGreen.releaseGreen, true);
});

test("computeVerdict: full-coverage classification — ratchets are drift, defects are hard", () => {
// Mirrors the expanded check set: the ratchets that historically surfaced in
// layers on the release PR (complexity/openapi/zizmor/…) are DRIFT → never block;
// the new real-defect gates (docs-all, integration) are HARD → block.
const results = [
{ id: "complexity", kind: "drift", ok: false },
{ id: "openapi-coverage", kind: "drift", ok: false },
{ id: "workflow-lint", kind: "drift", ok: false },
{ id: "dead-code", kind: "drift", ok: true },
{ id: "codeql-ratchet", kind: "drift", ok: true },
{ id: "docs-all", kind: "hard", ok: true },
{ id: "integration", kind: "hard", ok: true },
];
const v = computeVerdict(results);
// Three ratchets drifted but NONE block — release is still green, all reported.
assert.equal(v.releaseGreen, true);
assert.equal(v.drift.length, 3);

// A hard gate (integration assertion regression) flips it red.
const withHardFail = computeVerdict([...results, { id: "integration", kind: "hard", ok: false }]);
assert.equal(withHardFail.releaseGreen, false);
assert.equal(withHardFail.hardFailures.length, 1);
});
Loading