Repository navigation
ci: layered L0–L3 scaffold trust (replace stacked matrix) - #313
Conversation
Retire smoke-test and all-extensions stacking. Add registry/profile validation, per-template baselines, one-extension isolation, and curated profiles so green Actions imply real scaffolds work. Also fix file:// package resolution for nextjs/saas, webext types, turbo tsconfig paths, and lucide brand icon removals so L1 can pass. Closes #309 Closes #310 Closes #311 Closes #312 Co-authored-by: Cursor <cursoragent@cursor.com>
|
Warning Review limit reached
Next review available in: 52 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR replaces legacy smoke and stacked-combination workflows with layered L0–L3 CI. It adds registry-aware matrix and scaffold utilities, curated profiles, integrity validation, template updates, dependency adjustments, and revised CI documentation. ChangesLayered CI and template validation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant PullRequest
participant MatrixGenerator
participant LayerWorkflow
participant ScaffoldCheck
PullRequest->>MatrixGenerator: generate changed-only matrix
MatrixGenerator->>LayerWorkflow: publish matrix and count
LayerWorkflow->>ScaffoldCheck: run template, extension, or profile cell
ScaffoldCheck->>ScaffoldCheck: install and run applicable checks
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci-extensions.yml:
- Around line 79-81: Fix the Bash loop that builds ADDON_ARGS so empty addon
arrays do not produce invalid `for url in ; do` syntax. In
.github/workflows/ci-extensions.yml lines 79-81 and
.github/workflows/ci-profiles.yml lines 79-81, assign the joined matrix addons
value to an ADDONS variable first, then iterate over `$ADDONS` while preserving
the existing ADDON_ARGS construction.
- Around line 36-42: Prevent shell injection in the generation steps by adding
an env mapping for BASE_REF from github.base_ref and replacing direct
github.base_ref interpolation with $BASE_REF in
.github/workflows/ci-extensions.yml lines 36-42 and
.github/workflows/ci-profiles.yml lines 36-42; apply the same change to both
workflows while preserving their existing fetch and generate-matrix behavior.
In @.github/workflows/ci-templates.yml:
- Around line 58-78: The scaffold check job lacks failure diagnostics. Add a
subsequent step after “Scaffold check (template alone)” using a pinned
actions/upload-artifact action, configured with if: failure() and path `${{
runner.temp }}/cna-l1`, so scaffold logs, generated files, and metadata are
uploaded before runner cleanup.
In `@docs/TESTING.md`:
- Around line 51-53: Update the “Debug output” guidance to avoid claiming that
--verbose works with the shared run-scaffold-check.js runner; scope it
explicitly to direct CNA commands, unless run-scaffold-check.js is updated to
recognize and forward the option.
In `@scripts/ci/generate-matrix.js`:
- Around line 96-100: Update the loop over asTypes(extension.type) in the
matrix-generation flow to throw an actionable error when
canonicalTemplateDirForType(type) returns no directory or
findTemplateByDir(registry, canonicalDir) returns no template, instead of
silently continuing. Include the extension/type and missing mapping or template
context in the error, while preserving normal processing for valid templates so
every registered extension receives L2 coverage.
- Around line 45-54: Update changedPaths to fail the CI matrix generation when
the git diff command fails instead of returning an empty array. Preserve the
existing successful parsing of git diff output, but propagate or explicitly
surface the error so changed-only L2/L3 matrices cannot silently become empty
when the base ref or checkout is unavailable.
In `@scripts/ci/registry.js`:
- Around line 115-137: Update assertProfileValid to validate each resolved addon
against the selected template’s type before returning. Reject any addon whose
supported types do not include template.type, using the existing profile and
addon identifiers in the error; keep the category and incompatibility checks
intact.
In `@scripts/ci/run-scaffold-check.js`:
- Around line 189-207: Add a lint phase in the scaffold validation flow before
the type-check block, using hasScript(projectRoot, 'lint') and executing npm run
lint through the existing run helper so failures are reported per template and
phase. Keep the absent-script skip behavior consistent with type-check, and
preserve the existing format phase ordering.
In `@templates/nextjs-saas-ai-starter/package/index.js`:
- Line 1: Update resolvePackage to accept the full resolver signature (setup, {
appName, runCommand, usePnpm }) and use the selected package-manager inputs when
generating the db:push script instead of hardcoding pnpm. Preserve the existing
package JSON output and appName behavior.
In `@templates/nextjs-starter/package/index.js`:
- Around line 15-19: Align generated template Node engine requirements with
their dependencies: in templates/nextjs-starter/package/index.js, add
engines.node set to >=20.9.0 alongside the Next.js 16 dependencies; in
templates/turborepo-starter/package/index.js, raise engines.node to >=22.0.0 or
downgrade Vitest to a release supporting Node 18.18.0.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 45744039-135d-4b2a-af04-3e40a2de9fae
📒 Files selected for processing (40)
.github/workflows/ci-extensions.yml.github/workflows/ci-integrity.yml.github/workflows/ci-profiles.yml.github/workflows/ci-templates.yml.github/workflows/smoke-test.yml.github/workflows/test-combinations.yml.gitignoreREADME.mdci/profiles/nestjs-default.jsonci/profiles/nextjs-default.jsonci/profiles/react-default.jsonci/profiles/saas-default.jsonci/profiles/turborepo-default.jsonci/profiles/wdio-default.jsonci/profiles/webext-default.jsondocs/MAINTENANCE_CI.mddocs/MAINTENANCE_RUNBOOK.mddocs/MAINTENANCE_TEMPLATES.mddocs/TESTING.mdscripts/ci/generate-matrix.jsscripts/ci/registry.jsscripts/ci/run-scaffold-check.jsscripts/validate-templates.jstemplates/hono-starter/.prettierrc.cjstemplates/nextjs-saas-ai-starter/package.jsontemplates/nextjs-saas-ai-starter/package/index.jstemplates/nextjs-saas-ai-starter/src/features/admin/components/GitHubSettingsPanel.tsxtemplates/nextjs-saas-ai-starter/src/features/profile/components/ProfileClient.tsxtemplates/nextjs-saas-ai-starter/src/shared/components/ProfileCompletionWidget.tsxtemplates/nextjs-saas-ai-starter/src/shared/components/layout/nav/MyViewNav.tsxtemplates/nextjs-starter/.npmrctemplates/nextjs-starter/package.jsontemplates/nextjs-starter/package/index.jstemplates/turborepo-starter/apps/web/tsconfig.json.templatetemplates/turborepo-starter/package/index.jstemplates/turborepo-starter/packages/ui/tsconfig.jsontemplates/webextension-react-vite-starter/[src]/background/messageHandler.tstemplates/webextension-react-vite-starter/[src]/options/Options.test.tsxtemplates/webextension-react-vite-starter/tsconfig.json.templatetemplates/webextension-react-vite-starter/vitest.setup.ts.template
💤 Files with no reviewable changes (4)
- .github/workflows/test-combinations.yml
- templates/nextjs-starter/package.json
- .github/workflows/smoke-test.yml
- templates/nextjs-saas-ai-starter/package.json
| if [ "${{ github.event_name }}" = "pull_request" ]; then | ||
| BASE="origin/${{ github.base_ref }}" | ||
| git fetch origin "${{ github.base_ref }}" --depth=1 || true | ||
| node scripts/ci/generate-matrix.js --layer extensions --changed-only --base-ref "$BASE" | ||
| else | ||
| node scripts/ci/generate-matrix.js --layer extensions | ||
| fi |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Prevent script injection by passing GitHub context variables via env.
Directly interpolating ${{ github.base_ref }} in a shell script creates a template injection vulnerability. If a pull request targets a branch with a maliciously crafted name (e.g., main"; rm -rf /; #), the payload will be executed during script rendering. Map the value via the step's env object instead.
.github/workflows/ci-extensions.yml#L36-L42: Add anenvblock mappingBASE_REF: ${{ github.base_ref }}and use$BASE_REFin the bash script..github/workflows/ci-profiles.yml#L36-L42: Apply the exact same fix to the profiles generation step.
🔒️ Proposed fix (for `ci-extensions.yml`)
- name: Generate L2 matrix
id: set
+ env:
+ BASE_REF: ${{ github.base_ref }}
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
- BASE="origin/${{ github.base_ref }}"
- git fetch origin "${{ github.base_ref }}" --depth=1 || true
+ BASE="origin/$BASE_REF"
+ git fetch origin "$BASE_REF" --depth=1 || true
node scripts/ci/generate-matrix.js --layer extensions --changed-only --base-ref "$BASE"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if [ "${{ github.event_name }}" = "pull_request" ]; then | |
| BASE="origin/${{ github.base_ref }}" | |
| git fetch origin "${{ github.base_ref }}" --depth=1 || true | |
| node scripts/ci/generate-matrix.js --layer extensions --changed-only --base-ref "$BASE" | |
| else | |
| node scripts/ci/generate-matrix.js --layer extensions | |
| fi | |
| - name: Generate L2 matrix | |
| id: set | |
| env: | |
| BASE_REF: ${{ github.base_ref }} | |
| run: | | |
| if [ "${{ github.event_name }}" = "pull_request" ]; then | |
| BASE="origin/$BASE_REF" | |
| git fetch origin "$BASE_REF" --depth=1 || true | |
| node scripts/ci/generate-matrix.js --layer extensions --changed-only --base-ref "$BASE" | |
| else | |
| node scripts/ci/generate-matrix.js --layer extensions | |
| fi |
🧰 Tools
🪛 zizmor (1.26.1)
[error] 37-37: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
[error] 38-38: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
📍 Affects 2 files
.github/workflows/ci-extensions.yml#L36-L42(this comment).github/workflows/ci-profiles.yml#L36-L42
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci-extensions.yml around lines 36 - 42, Prevent shell
injection in the generation steps by adding an env mapping for BASE_REF from
github.base_ref and replacing direct github.base_ref interpolation with
$BASE_REF in .github/workflows/ci-extensions.yml lines 36-42 and
.github/workflows/ci-profiles.yml lines 36-42; apply the same change to both
workflows while preserving their existing fetch and generate-matrix behavior.
Source: Linters/SAST tools
| for url in ${{ join(matrix.cell.addons, ' ') }}; do | ||
| ADDON_ARGS+=(--addon-url "$url") | ||
| done |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Fix bash syntax error when the addons array is empty.
If a matrix cell resolves to an empty addons array (which could happen if a curated profile is tested without addons), ${{ join(matrix.cell.addons, ' ') }} renders as an empty string. The resulting script becomes for url in ; do, which triggers a parse-time syntax error in Bash and instantly fails the CI job. Evaluate the joined string into a Bash variable first to gracefully handle empty arrays.
.github/workflows/ci-extensions.yml#L79-L81: Extract the joined string toADDONS="${{ join(matrix.cell.addons, ' ') }}"and loop over$ADDONS..github/workflows/ci-profiles.yml#L79-L81: Apply the exact same bash loop syntax fix.
💚 Proposed fix (apply to both files)
- for url in ${{ join(matrix.cell.addons, ' ') }}; do
+ ADDONS="${{ join(matrix.cell.addons, ' ') }}"
+ for url in $ADDONS; do
ADDON_ARGS+=(--addon-url "$url")
done📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for url in ${{ join(matrix.cell.addons, ' ') }}; do | |
| ADDON_ARGS+=(--addon-url "$url") | |
| done | |
| ADDONS="${{ join(matrix.cell.addons, ' ') }}" | |
| for url in $ADDONS; do | |
| ADDON_ARGS+=(--addon-url "$url") | |
| done |
📍 Affects 2 files
.github/workflows/ci-extensions.yml#L79-L81(this comment).github/workflows/ci-profiles.yml#L79-L81
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci-extensions.yml around lines 79 - 81, Fix the Bash loop
that builds ADDON_ARGS so empty addon arrays do not produce invalid `for url in
; do` syntax. In .github/workflows/ci-extensions.yml lines 79-81 and
.github/workflows/ci-profiles.yml lines 79-81, assign the joined matrix addons
value to an ADDONS variable first, then iterate over `$ADDONS` while preserving
the existing ADDON_ARGS construction.
| - name: Scaffold check (template alone) | ||
| env: | ||
| NODE_OPTIONS: --max-old-space-size=8192 | ||
| run: | | ||
| EXTRA=() | ||
| # Turborepo needs an explicit scope for non-interactive scaffolds. | ||
| if [ "${{ matrix.cell.templateDir }}" = "turborepo-starter" ]; then | ||
| EXTRA+=(--set "scope=@app/" --set "projectName=turbo-app") | ||
| fi | ||
| # WDIO browser tests are not reliable without a dedicated runner image. | ||
| if [ "${{ matrix.cell.templateDir }}" = "wdio-starter" ]; then | ||
| EXTRA+=(--skip-test) | ||
| fi | ||
| # SaaS: service-backed tests/build; L1 covers install + type-check. | ||
| if [ "${{ matrix.cell.templateDir }}" = "nextjs-saas-ai-starter" ]; then | ||
| EXTRA+=(--skip-test --skip-build) | ||
| fi | ||
| node scripts/ci/run-scaffold-check.js \ | ||
| --template-url "${{ matrix.cell.templateUrl }}" \ | ||
| --workdir "${{ runner.temp }}/cna-l1" \ | ||
| "${EXTRA[@]}" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Upload the failed scaffold directory before the runner is destroyed.
The scaffold runner preserves its work directory on failure, but this workflow has no if: failure() artifact step. Add a pinned actions/upload-artifact step for ${{ runner.temp }}/cna-l1 so generated files and metadata remain available for diagnosis.
As per PR objectives, scaffold logs and generated metadata must be available on failure.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci-templates.yml around lines 58 - 78, The scaffold check
job lacks failure diagnostics. Add a subsequent step after “Scaffold check
(template alone)” using a pinned actions/upload-artifact action, configured with
if: failure() and path `${{ runner.temp }}/cna-l1`, so scaffold logs, generated
files, and metadata are uploaded before runner cleanup.
| ### Debug output | ||
|
|
||
| Add `--verbose` to any command to see detailed scaffolding logs. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not claim that --verbose works with the shared runner.
run-scaffold-check.js ignores unknown options and does not forward --verbose. Either scope this instruction to direct CNA commands or add explicit forwarding in the runner.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/TESTING.md` around lines 51 - 53, Update the “Debug output” guidance to
avoid claiming that --verbose works with the shared run-scaffold-check.js
runner; scope it explicitly to direct CNA commands, unless run-scaffold-check.js
is updated to recognize and forward the option.
| function changedPaths(baseRef) { | ||
| try { | ||
| const out = execSync(`git diff --name-only ${baseRef}...HEAD`, { | ||
| cwd: REPO_ROOT, | ||
| encoding: 'utf8', | ||
| }); | ||
| return out.split('\n').map((s) => s.trim()).filter(Boolean); | ||
| } catch { | ||
| return []; | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fail closed when the changed-path diff cannot be generated.
Returning [] on any Git error makes changed-only L2/L3 matrices silently empty. A shallow checkout or missing base ref could therefore produce green CI with no cells.
Proposed fix
- } catch {
- return [];
+ } catch (error) {
+ throw new Error(
+ `Unable to determine changed paths against "${baseRef}"`,
+ { cause: error },
+ );
}As per PR objectives, changed extensions and affected profiles must receive PR coverage.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| function changedPaths(baseRef) { | |
| try { | |
| const out = execSync(`git diff --name-only ${baseRef}...HEAD`, { | |
| cwd: REPO_ROOT, | |
| encoding: 'utf8', | |
| }); | |
| return out.split('\n').map((s) => s.trim()).filter(Boolean); | |
| } catch { | |
| return []; | |
| } | |
| function changedPaths(baseRef) { | |
| try { | |
| const out = execSync(`git diff --name-only ${baseRef}...HEAD`, { | |
| cwd: REPO_ROOT, | |
| encoding: 'utf8', | |
| }); | |
| return out.split('\n').map((s) => s.trim()).filter(Boolean); | |
| } catch (error) { | |
| throw new Error( | |
| `Unable to determine changed paths against "${baseRef}"`, | |
| { cause: error }, | |
| ); | |
| } |
🧰 Tools
🪛 OpenGrep (1.25.0)
[ERROR] 47-50: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.
(coderabbit.command-injection.exec-js)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ci/generate-matrix.js` around lines 45 - 54, Update changedPaths to
fail the CI matrix generation when the git diff command fails instead of
returning an empty array. Preserve the existing successful parsing of git diff
output, but propagate or explicitly surface the error so changed-only L2/L3
matrices cannot silently become empty when the base ref or checkout is
unavailable.
| for (const type of asTypes(extension.type)) { | ||
| const canonicalDir = canonicalTemplateDirForType(type); | ||
| if (!canonicalDir) continue; | ||
| const template = findTemplateByDir(registry, canonicalDir); | ||
| if (!template) continue; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not silently omit extensions without a canonical L2 template.
Both missing mappings and missing registry templates currently continue, so an extension can be registered yet receive zero L2 cells. Throw an actionable error instead.
Proposed fix
const canonicalDir = canonicalTemplateDirForType(type);
- if (!canonicalDir) continue;
+ if (!canonicalDir) {
+ throw new Error(`No canonical template configured for extension type "${type}"`);
+ }
const template = findTemplateByDir(registry, canonicalDir);
- if (!template) continue;
+ if (!template) {
+ throw new Error(`Canonical template directory "${canonicalDir}" is not registered`);
+ }As per PR objectives, L2 must test every registered extension independently.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for (const type of asTypes(extension.type)) { | |
| const canonicalDir = canonicalTemplateDirForType(type); | |
| if (!canonicalDir) continue; | |
| const template = findTemplateByDir(registry, canonicalDir); | |
| if (!template) continue; | |
| for (const type of asTypes(extension.type)) { | |
| const canonicalDir = canonicalTemplateDirForType(type); | |
| if (!canonicalDir) { | |
| throw new Error(`No canonical template configured for extension type "${type}"`); | |
| } | |
| const template = findTemplateByDir(registry, canonicalDir); | |
| if (!template) { | |
| throw new Error(`Canonical template directory "${canonicalDir}" is not registered`); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ci/generate-matrix.js` around lines 96 - 100, Update the loop over
asTypes(extension.type) in the matrix-generation flow to throw an actionable
error when canonicalTemplateDirForType(type) returns no directory or
findTemplateByDir(registry, canonicalDir) returns no template, instead of
silently continuing. Include the extension/type and missing mapping or template
context in the error, while preserving normal processing for valid templates so
every registered extension receives L2 coverage.
| function assertProfileValid(registry, profile) { | ||
| const template = registry.templates.find((t) => templateDir(t) === profile.templateDir); | ||
| if (!template) { | ||
| throw new Error(`Profile ${profile.id}: unknown templateDir "${profile.templateDir}"`); | ||
| } | ||
| const addons = resolveProfileAddons(registry, profile); | ||
| const categories = new Map(); | ||
| for (const ext of addons) { | ||
| if (categories.has(ext.category)) { | ||
| throw new Error( | ||
| `Profile ${profile.id}: category "${ext.category}" used twice (${categories.get(ext.category)} and ${ext.slug})`, | ||
| ); | ||
| } | ||
| categories.set(ext.category, ext.slug); | ||
| } | ||
| const selected = []; | ||
| for (const ext of addons) { | ||
| if (hasIncompatibility(selected, ext)) { | ||
| throw new Error(`Profile ${profile.id}: incompatible addons involving ${ext.slug}`); | ||
| } | ||
| selected.push(ext); | ||
| } | ||
| return { template, addons }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Reject profile addons that do not support the selected template type.
The validator currently accepts any known addon regardless of its type, so an invalid template/addon composition can pass L0 and reach L3.
Proposed compatibility check
const addons = resolveProfileAddons(registry, profile);
+ const templateTypes = new Set(asTypes(template.type));
+ for (const ext of addons) {
+ if (!asTypes(ext.type).some((type) => templateTypes.has(type))) {
+ throw new Error(
+ `Profile ${profile.id}: addon "${ext.slug}" is incompatible with template "${profile.templateDir}"`,
+ );
+ }
+ }
const categories = new Map();📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| function assertProfileValid(registry, profile) { | |
| const template = registry.templates.find((t) => templateDir(t) === profile.templateDir); | |
| if (!template) { | |
| throw new Error(`Profile ${profile.id}: unknown templateDir "${profile.templateDir}"`); | |
| } | |
| const addons = resolveProfileAddons(registry, profile); | |
| const categories = new Map(); | |
| for (const ext of addons) { | |
| if (categories.has(ext.category)) { | |
| throw new Error( | |
| `Profile ${profile.id}: category "${ext.category}" used twice (${categories.get(ext.category)} and ${ext.slug})`, | |
| ); | |
| } | |
| categories.set(ext.category, ext.slug); | |
| } | |
| const selected = []; | |
| for (const ext of addons) { | |
| if (hasIncompatibility(selected, ext)) { | |
| throw new Error(`Profile ${profile.id}: incompatible addons involving ${ext.slug}`); | |
| } | |
| selected.push(ext); | |
| } | |
| return { template, addons }; | |
| function assertProfileValid(registry, profile) { | |
| const template = registry.templates.find((t) => templateDir(t) === profile.templateDir); | |
| if (!template) { | |
| throw new Error(`Profile ${profile.id}: unknown templateDir "${profile.templateDir}"`); | |
| } | |
| const addons = resolveProfileAddons(registry, profile); | |
| const templateTypes = new Set(asTypes(template.type)); | |
| for (const ext of addons) { | |
| if (!asTypes(ext.type).some((type) => templateTypes.has(type))) { | |
| throw new Error( | |
| `Profile ${profile.id}: addon "${ext.slug}" is incompatible with template "${profile.templateDir}"`, | |
| ); | |
| } | |
| } | |
| const categories = new Map(); | |
| for (const ext of addons) { | |
| if (categories.has(ext.category)) { | |
| throw new Error( | |
| `Profile ${profile.id}: category "${ext.category}" used twice (${categories.get(ext.category)} and ${ext.slug})`, | |
| ); | |
| } | |
| categories.set(ext.category, ext.slug); | |
| } | |
| const selected = []; | |
| for (const ext of addons) { | |
| if (hasIncompatibility(selected, ext)) { | |
| throw new Error(`Profile ${profile.id}: incompatible addons involving ${ext.slug}`); | |
| } | |
| selected.push(ext); | |
| } | |
| return { template, addons }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ci/registry.js` around lines 115 - 137, Update assertProfileValid to
validate each resolved addon against the selected template’s type before
returning. Reject any addon whose supported types do not include template.type,
using the existing profile and addon identifiers in the error; keep the category
and incompatibility checks intact.
| if (hasScript(projectRoot, 'format')) { | ||
| console.log(`\n▶ [format] npm run format --if-present (soft)`); | ||
| const formatResult = spawnSync('npm', ['run', 'format', '--if-present'], { | ||
| cwd: projectRoot, | ||
| stdio: 'inherit', | ||
| env: { ...process.env, CI: 'true', FORCE_COLOR: '0' }, | ||
| }); | ||
| if (formatResult.status !== 0) { | ||
| console.warn( | ||
| `⚠ [format] failed with exit ${formatResult.status} — continuing (format is soft in CI)`, | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| if (hasScript(projectRoot, 'type-check')) { | ||
| run('type-check', 'npm', ['run', 'type-check'], { cwd: projectRoot }); | ||
| } else { | ||
| console.log('ℹ [type-check] skipped (script not present)'); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Run the template’s lint script before type checking.
lint counts toward the non-empty guard but is never executed, allowing lint-broken templates to pass L1.
Proposed fix
if (hasScript(projectRoot, 'format')) {
// ...
}
+ if (hasScript(projectRoot, 'lint')) {
+ run('lint', 'npm', ['run', 'lint'], { cwd: projectRoot });
+ } else {
+ console.log('ℹ [lint] skipped (script not present)');
+ }
+
if (hasScript(projectRoot, 'type-check')) {As per PR objectives, L1 must validate every template independently with clear phase failures.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (hasScript(projectRoot, 'format')) { | |
| console.log(`\n▶ [format] npm run format --if-present (soft)`); | |
| const formatResult = spawnSync('npm', ['run', 'format', '--if-present'], { | |
| cwd: projectRoot, | |
| stdio: 'inherit', | |
| env: { ...process.env, CI: 'true', FORCE_COLOR: '0' }, | |
| }); | |
| if (formatResult.status !== 0) { | |
| console.warn( | |
| `⚠ [format] failed with exit ${formatResult.status} — continuing (format is soft in CI)`, | |
| ); | |
| } | |
| } | |
| if (hasScript(projectRoot, 'type-check')) { | |
| run('type-check', 'npm', ['run', 'type-check'], { cwd: projectRoot }); | |
| } else { | |
| console.log('ℹ [type-check] skipped (script not present)'); | |
| } | |
| if (hasScript(projectRoot, 'format')) { | |
| console.log(`\n▶ [format] npm run format --if-present (soft)`); | |
| const formatResult = spawnSync('npm', ['run', 'format', '--if-present'], { | |
| cwd: projectRoot, | |
| stdio: 'inherit', | |
| env: { ...process.env, CI: 'true', FORCE_COLOR: '0' }, | |
| }); | |
| if (formatResult.status !== 0) { | |
| console.warn( | |
| `⚠ [format] failed with exit ${formatResult.status} — continuing (format is soft in CI)`, | |
| ); | |
| } | |
| } | |
| if (hasScript(projectRoot, 'lint')) { | |
| run('lint', 'npm', ['run', 'lint'], { cwd: projectRoot }); | |
| } else { | |
| console.log('ℹ [lint] skipped (script not present)'); | |
| } | |
| if (hasScript(projectRoot, 'type-check')) { | |
| run('type-check', 'npm', ['run', 'type-check'], { cwd: projectRoot }); | |
| } else { | |
| console.log('ℹ [type-check] skipped (script not present)'); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/ci/run-scaffold-check.js` around lines 189 - 207, Add a lint phase in
the scaffold validation flow before the type-check block, using
hasScript(projectRoot, 'lint') and executing npm run lint through the existing
run helper so failures are reported per template and phase. Keep the
absent-script skip behavior consistent with type-check, and preserve the
existing format phase ordering.
| @@ -0,0 +1,163 @@ | |||
| module.exports = function resolvePackage(_setup, { appName }) { | |||
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Honor the package-resolver contract instead of hardcoding pnpm.
The omitted runCommand/usePnpm inputs leave db:push hardcoded to pnpm, so non-pnpm scaffolds cannot run this script. Accept the full resolver arguments and generate the command for the selected package manager.
As per coding guidelines, **/templates/**/package/index.js must export (setup, { appName, runCommand, usePnpm }) => packageJson.
Also applies to: 22-22
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@templates/nextjs-saas-ai-starter/package/index.js` at line 1, Update
resolvePackage to accept the full resolver signature (setup, { appName,
runCommand, usePnpm }) and use the selected package-manager inputs when
generating the db:push script instead of hardcoding pnpm. Preserve the existing
package JSON output and appName behavior.
Source: Coding guidelines
| dependencies: { | ||
| "next": "^16.2.10", | ||
| "react": "^19.2.7", | ||
| "react-dom": "^19.2.7", | ||
| "zod": "^4.4.3" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
git ls-files 'templates/**/package/index.js' 'templates/**/package.json' 'templates/**/package/*.js' | sed -n '1,200p'Repository: Create-Node-App/cna-templates
Length of output: 1312
🏁 Script executed:
sed -n '1,220p' templates/nextjs-starter/package/index.js
printf '\n---\n'
sed -n '1,220p' templates/turborepo-starter/package/index.js
printf '\n---\n'
git ls-files 'templates/**/package/*.js' | xargs -I{} sh -c 'printf "\n### %s\n" "{}"; sed -n "1,220p" "{}" | grep -n "engines\\|node\\|vitest\\|next"'Repository: Create-Node-App/cna-templates
Length of output: 6970
🌐 Web query:
Next.js 16.2.10 engines node requirement and Vitest 4.1.9 engines node requirement
💡 Result:
The Node.js engine requirements for the specified versions are as follows: Next.js 16.2.10 requires Node.js 20.9.0 or later [1][2]. Vitest 4.1.9 requires Node.js 22.0.0 or later [3][4]. While Vitest 4.0 initially introduced a requirement for Node.js 20.0.0+ [5], subsequent updates—specifically starting with the requirement for Vite 6.4 in later 4.x releases—increased the minimum Node.js version requirement to 22 [6][4].
Citations:
- 1: https://nextjs.org/docs/app/guides/upgrading/version-16
- 2: https://npmx.dev/package/next/v/%5E16
- 3: https://npmx.dev/package/vitest/v/4.1.9
- 4: https://github.com/vitest-dev/vitest/releases
- 5: https://vitest.dev/guide/migration.html
- 6: feat!: require Node.js 22 and Vite 6.4 vitest-dev/vitest#10178
Align the generated Node engine ranges with the dependencies they ship.
templates/nextjs-starter/package/index.js#L15-L19: addengines.node: ">=20.9.0"for Next.js 16.templates/turborepo-starter/package/index.js#L32-L44: raiseengines.nodeto>=22.0.0or downgrade Vitest to a version that supports Node 18.18.0.
📍 Affects 2 files
templates/nextjs-starter/package/index.js#L15-L19(this comment)templates/turborepo-starter/package/index.js#L32-L32
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@templates/nextjs-starter/package/index.js` around lines 15 - 19, Align
generated template Node engine requirements with their dependencies: in
templates/nextjs-starter/package/index.js, add engines.node set to >=20.9.0
alongside the Next.js 16 dependencies; in
templates/turborepo-starter/package/index.js, raise engines.node to >=22.0.0 or
downgrade Vitest to a release supporting Node 18.18.0.
Source: MCP tools
The wdio and saas-ai templates have known service-/runner-dependent tests that L1 already skips (--skip-test, and --skip-build for saas). Apply the same skips when those templates are used as the canonical base in L2 (extension isolation) and L3 (curated profiles) so tooling extensions aren't failed by the base template's own unrunnable tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause of remaining L2/L3 red on nextjs extensionsThe new layered CI surfaced a real, previously-hidden bug: dependencies declared in an extension's Traced to an upstream bug in
This commit (Cause 2)Aligned L2/L3 test skips with L1 for the |
- nextjs-tailwindcss: bump tailwindcss to ^4 to match @tailwindcss/postcss and the @import "tailwindcss" CSS entry (was pinned to v3, so build failed with Can't resolve 'tailwindcss'). - nextjs-prisma: pin prisma/@prisma/client to ^6 (schema still uses datasource url, which Prisma 7 rejects) and add postinstall prisma generate so PrismaClient exists for type-check after install. Co-authored-by: Cursor <cursoragent@cursor.com>
strictPropertyInitialization fails type-check because db is assigned in onModuleInit rather than the constructor. Use definite assignment (!). Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
Summary
Implements #309 (and #310–#312): replace false-green smoke + all-extensions stacking with layered CI that attributes failures.
ci-integrity.ymlci-templates.ymlfile://dirs)ci-extensions.ymlci-profiles.ymlci/profiles/*.jsonRemoved:
smoke-test.yml(empty scaffolds on wrong slug paths) andtest-combinations.yml(random + 18–32 addon mega-stacks).Template fixes required for honest L1
package/modules (CNA ignores staticpackage.jsononfile://; coexist withpackage/breaks resolver).npmrcfor peer resolutiontsconfigpaths for@app/ui, exclude UI tests fromtsc, rootvitestGitBranch/Link2Test plan
node scripts/validate-templates.jsnode scripts/ci/generate-matrix.js --layer validate-profilesCloses #309
Closes #310
Closes #311
Closes #312
Made with Cursor
Summary by CodeRabbit