You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
production build pass (40/40 pages; existing warnings only)
Vitest 1246 passed / 12 skipped
Mastra gate pass
critical npm audit gate pass
git diff --check: pass
Recovery review
The preserved Task 108 recovery branches were reviewed separately. They are intentionally not merged wholesale because they contain stale Supabase counts/config and older skill ownership guidance that conflicts with current main/PR #52.
Summary by Sourcery
Add automated DeepSeek routing certification for representative skill ownership and high-risk verification decisions.
New Features:
Add a repeatable DeepSeek routing certification harness covering six representative skill-routing and verifier scenarios.
Enhancements:
Add fail-closed parsing and exact owner/verifier evaluation for routing certification results.
Tests:
Add Vitest coverage for final-answer parsing, exact routing decisions, and aggregated probe failures.
This PR introduces a repeatable DeepSeek routing certification workflow for the PR #52 skill-routing architecture, combining live headless CLI probes for six owner and S4-verifier scenarios with lightweight Vitest source guards that prevent required cases or fail-closed handling from being removed.
Sequence diagram for DeepSeek routing certification
sequenceDiagram
participant Harness as verify-deepseek-routing.mjs
participant DeepSeekCLI as DeepSeek headless CLI
participant Vitest as Vitest guard
loop six routing cases
Harness->>DeepSeekCLI: spawnSync npx @deepseek-ai/dsh
DeepSeekCLI-->>Harness: stdout, stderr, exit status
Harness->>Harness: expected.test(output)
alt output matches and status is zero
Harness-->>Harness: log PASS
else probe fails
Harness->>Harness: process.exitCode = 1
Harness-->>Harness: log expected output
end
end
Vitest->>Vitest: readFileSync script source
Vitest->>Vitest: assert required cases and task-verifier
Vitest->>Vitest: assert process.exitCode = 1 and expected handling
Loading
File-Level Changes
Change
Details
Files
Adds a six-case DeepSeek CLI certification harness that validates routing owners and S4 verifier requirements through exit status and regex-matched output.
Defines probes for CopilotKit, unknown failures, Supabase RLS S4, Mastra, Gemini, and Stripe payment S4.
Runs each probe through the headless DeepSeek Harness with a two-minute timeout and inherited environment.
Reports per-case results and fails the process if any probe exits unsuccessfully or misses its expected routing output.
scripts/verify-deepseek-routing.mjs
Adds Vitest guards to ensure the certification harness retains its required coverage and fail-closed behavior.
Checks that all six routing domains and task-verifier probes remain defined.
Checks that mismatches set a nonzero process exit code and produce expected-output diagnostics.
src/lib/deepseek/verify-deepseek-routing.test.ts
Tips and commands
Interacting with Sourcery
Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment @sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment @sourcery-ai review to trigger a new review!
The reason will be displayed to describe this comment to others. Learn more.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments### Comment 1
<locationpath="scripts/verify-deepseek-routing.mjs"line_range="5-10" />
<code_context>
+['copilotkit', 'Fix a CopilotKit generative UI rendering bug. Answer only with the single MDE skill owner.', /\bcopilotkit\b/i],
</code_context>
<issue_to_address>
**issue (bug_risk):** The certification accepts any occurrence of the expected skill name in combined stdout/stderr, rather than verifying the routed owner or verifier decision. When `dsh` echoes the submitted prompt or includes it in diagnostics, five probes pass even if routing fails because those prompts already contain their expected terms.
**Triggers:** When the DeepSeek CLI echoes the input prompt or includes request text in a warning/error.
**Suggested fix:** Parse the structured routing result, or require the expected owner and verifier decision in the answer while rejecting echoed prompt/diagnostic text.
</issue_to_address>
### Comment 2
<locationpath="scripts/verify-deepseek-routing.mjs"line_range="15-20" />
<code_context>
+
+let failed = 0;
+for (const [name, prompt, expected] of cases) {
+ const result = spawnSync('npx', ['--yes', '@deepseek-ai/dsh', '--profile', 'headless', prompt], {
+ cwd: process.cwd(),
+ encoding: 'utf8',
+ timeout: 120000,
+ env: process.env,
+ });
+ const output = `${result.stdout ?? ''}\n${result.stderr ?? ''}`;
+ const ok = result.status === 0 && expected.test(output);
</code_context>
<issue_to_address>
**issue (broader_impact):** Each certification run resolves `@deepseek-ai/dsh` from the network without a version constraint or lockfile entry, so the same repository commit can execute different CLI versions and produce different certification results.
**Triggers:** When npm resolves a newer package release, or the package registry/cache changes between runs.
**Suggested fix:** Pin the CLI version and manage it as a declared dependency, or invoke a repository-controlled executable with a locked dependency tree.
</issue_to_address>
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
The PR successfully implements the DeepSeek routing certification harness and the fail-closed Vitest guard as required by the acceptance criteria. While the functional requirements are met, there are significant implementation concerns regarding the reliability and performance of the certification script and the quality of the associated test suite.
The current use of spawnSync without shell: true will break the script on Windows environments. Furthermore, the sequential execution of probes results in inefficient CI/CD runtimes. The Vitest suite currently employs 'meta-testing'—asserting on the script's source code strings rather than its behavior—which provides low functional assurance and high maintenance overhead. These issues should be addressed to ensure a robust and cross-platform certification process.
About this PR
The use of spawnSync('npx', ...) without shell: true will cause the certification script to fail in Windows environments as npx is not a standalone binary.
The certification script executes all DeepSeek probes sequentially. With 6 cases each potentially taking up to 120 seconds, this could result in a 12-minute execution time, which is inefficient for the CI/CD pipeline.
Test suggestions
Verify that the certification script executes and validates the CopilotKit, Gemini, and Mastra skill owners using regex expectations.
Verify that the certification script validates the 'task-verifier' requirement for Supabase and Stripe S4 scenarios.
Verify that a failure in any DeepSeek probe sets process.exitCode = 1 to fail the CI/CD pipeline.
Verify that the Vitest suite correctly identifies if a mandatory routing probe is missing from the certification script.
Implement functional testing using vi.mock for child_process to verify the certification logic without relying on static source code analysis.
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Implement functional testing using `vi.mock` for child_process to verify the certification logic without relying on static source code analysis.
The reason will be displayed to describe this comment to others. Learn more.
🔥 The Roast: The previous fix removed the broad regex but put the answer key in every prompt. This fallback then treats any echoed owner=... verifier=... line as the final answer, so a probe can pass without routing at all.
🩹 The Fix: Keep expected values out of the prompts, accept only a structured answer from an explicit final-answer boundary, and add a regression test that echoes the exact prompt.
📏 Severity: warning
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
The reason will be displayed to describe this comment to others. Learn more.
🔥 The Roast: The two-minute timeout sends SIGTERM and then politely waits forever if the CLI ignores it. That is less a timeout than a strongly worded suggestion.
🩹 The Fix: Schedule a forced SIGKILL after a short grace period, or use a child-process timeout helper that guarantees termination, and cover a non-cooperative child with a test.
📏 Severity: warning
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
The reason will be displayed to describe this comment to others. Learn more.
🔥 The Roast: The guard for “required routing cases” was refactored so hard it disappeared. These tests cover parsing and aggregation, but never assert that cases still contains all six owners and verifier decisions; an empty matrix can still become PASS (0/0).
🩹 The Fix: Assert the complete six-case matrix, including each verifier flag, make runCases reject an empty probe list, and add a zero-case regression test.
📏 Severity: warning
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Task 111 · DeepSeek Routing Certification
Adds a small, repeatable certification harness for the merged PR #52 skill-routing architecture when used by DeepSeek Harness.
What changed
scripts/verify-deepseek-routing.mjsVerification
npm run floor: passgit diff --check: passRecovery review
The preserved Task 108 recovery branches were reviewed separately. They are intentionally not merged wholesale because they contain stale Supabase counts/config and older skill ownership guidance that conflicts with current main/PR #52.
Summary by Sourcery
Add automated DeepSeek routing certification for representative skill ownership and high-risk verification decisions.
New Features:
Enhancements:
Tests: