Repository navigation
ci(quality): wire Stryker mutation testing as advisory nightly (Fase 7 Task 11) - #3898
Conversation
…7 Task 11)
Wire Stryker mutation testing over the 8 critical modules as an ADVISORY,
NIGHTLY-ONLY job. It never runs on PRs and the mutation score is not yet
enforced as a ratchet (that lands in a later INT phase) — for now the nightly
just runs and uploads the HTML/JSON report as an artifact.
Test runner — @stryker-mutator/tap-runner (NOT vitest)
------------------------------------------------------
The 8 modules (chatCore, combo, accountFallback, src/sse/services/auth,
routeGuard, error, publicCreds, circuitBreaker) are covered by node:test files
in tests/unit/ (run via `node --import tsx --test`), NOT by vitest. The vitest
config only includes a small set of .test.tsx + open-sse/**/__tests__ files, so
@stryker-mutator/vitest-runner would find ZERO covering tests for them. Switched
testRunner to "tap" and configured tap.nodeArgs to load tsx + the project
polyfill exactly as `npm run test:unit` does, so each node:test file runs under
the runner and its TAP output is parsed.
Key calibration findings (each fixed):
- ignorePatterns must NOT exclude tests/ — Stryker copies the project to a
sandbox and runs the runner there; excluding tests means the tap glob finds
nothing ("No tests were found"). Scoped ignorePatterns to heavy, irrelevant
trees only (build output, coverage, docs/i18n, other worktrees).
- tap.testFiles enumerates the explicit covering union (files importing any of
the 8 modules), not a broad glob, so the dry-run stays tractable. A glob over
the full ~1300-file unit suite would make the per-test dry-run take hours.
- Pruned 11 files from the union: 7 timing/heap/streaming-sensitive integration
tests (flake under the sandbox) and 5 source-introspection tests that
readFileSync the source of a mutated module and assert on its content (those
are incompatible with mutation by design — the sandbox content is mutated).
- concurrency=1 + dryRunTimeoutMinutes=30 for a deterministic baseline.
Validated (NOT a faked run)
---------------------------
- `npx stryker run --dryRunOnly` exits 0: all 129 covering test files run
GREEN in the Stryker sandbox and the perTest coverage map builds for all 8
instrumented modules (15,261 mutants). Baseline dry-run ~20 min at
concurrency=1; the mutation phase runs on top. So the nightly produces REAL
mutation scores for the 8 modules.
- A narrowed end-to-end run (error.ts + 2 covering files) reached 61% with
mutants killed/surviving, confirming scoring works.
- `npm run check:deps` and `check-lockfile` accept the two new devDeps.
- nightly-mutation.yml parses as valid YAML.
Changes
-------
- devDeps @stryker-mutator/core + @stryker-mutator/tap-runner (real npm install)
- dependency-allowlist.json: both packages added
- package.json: "test:mutation": "stryker run"
- stryker.conf.json: tap runner + 129-file covering set + 8-module mutate
- .github/workflows/nightly-mutation.yml: schedule "17 3 * * *" + workflow_dispatch,
advisory (continue-on-error), npm ci + npx stryker run, upload reports/mutation/
- .gitignore: ignore .stryker-tmp/, reports/mutation/, stryker-output-*.json
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Stryker mutation testing to the project, adding the core and TAP runner packages, updating the dependency allowlist, and configuring stryker.conf.json to run tests sequentially with the TAP runner. Feedback points out a style guide violation because stryker.conf.json is placed in the project root instead of config/quality/. Additionally, there is a concern that running with a concurrency of one for over 15,000 mutants will cause the nightly CI workflow to time out, suggesting that database state should be isolated to allow parallel execution.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| "test:protocols:e2e": "node scripts/dev/run-protocol-clients-tests.mjs", | ||
| "test:vitest": "vitest run --config vitest.mcp.config.ts", | ||
| "test:vitest:ui": "vitest run --config vitest.config.ts tests/unit/ui", | ||
| "test:mutation": "stryker run", |
There was a problem hiding this comment.
De acordo com o Guia de Estilo do Repositório (Regra 1 - File Placement & Organization), o diretório raiz do projeto deve conter apenas os arquivos de configuração explicitamente listados. Como o stryker.conf.json não está nessa lista, ele deve ser movido para config/quality/stryker.conf.json para manter a raiz limpa.
Podemos ajustar o script test:mutation para apontar para o novo caminho do arquivo de configuração.
| "test:mutation": "stryker run", | |
| "test:mutation": "stryker run config/quality/stryker.conf.json", |
References
- O diretório raiz do projeto deve conter apenas arquivos de configuração específicos listados no guia de estilo. Arquivos de configuração adicionais, como o do Stryker, devem ser colocados em subdiretórios apropriados (ex:
config/quality/). (link)
| "timeoutMS": 60000, | ||
| "timeoutFactor": 2.5, | ||
| "concurrency": 2, | ||
| "concurrency": 1, |
There was a problem hiding this comment.
Definir "concurrency": 1 garante um baseline determinístico ao evitar concorrência em testes que compartilham estado do SQLite ou do módulo. No entanto, com 15.261 mutantes, a execução sequencial será extremamente lenta e excederá facilmente o limite de 120 minutos do workflow do GitHub Actions, resultando em relatórios de mutação muito parciais e incompletos.
Para viabilizar a execução paralela e obter relatórios completos no orçamento de tempo do nightly, considere isolar o estado do banco de dados em cada arquivo de teste (por exemplo, utilizando bancos de dados SQLite em memória ou arquivos temporários exclusivos por execução de teste).
…7 Task 11) (cheaper-inference#3898) Integrated into release/v3.8.26
O quê
Liga o Stryker (mutation testing) como job nightly advisory sobre os 8 módulos críticos — fecha o gap #2 da auditoria de completude (config existia mas estava órfã, sem script/job).
Decisão técnica: runner = tap-runner (não vitest)
Os 8 módulos críticos (chatCore/combo/accountFallback/auth/routeGuard/error/publicCreds/circuitBreaker) são cobertos por node:test, não vitest (vitest cobre só MCP/autoCombo/cache). Com vitest-runner o Stryker acharia zero testes cobrindo. Config ajustada para
@stryker-mutator/tap-runnerrodando os testes node:test via tsx.Mudanças
stryker.conf.json: runner tap + comando node:test (tsx + polyfill);mutate= 8 módulos;ignorePatternscorrigido (não excluirtests/— o Stryker sandboxa);tap.testFiles= união explícita das 129 specs que cobrem os 8 módulos (11 podadas: timing/heap/streaming-sensitive + introspecção-de-source incompatível com mutação por design).package.json: devDeps@stryker-mutator/core+@stryker-mutator/tap-runner+ scripttest:mutation..github/workflows/nightly-mutation.yml: schedule17 3 * * *(cron livre) + workflow_dispatch; advisory (continue-on-error);timeout-minutes: 120; upload do report. Não roda em PR.config/quality/dependency-allowlist.json: + 2 entradas stryker (o gate check-deps exige)..gitignore: artefatos.stryker-tmp/,reports/mutation/.Validação
npx stryker --dryRunOnly):Initial test run succeeded. Ran 129 tests→ instrumentou8 source file(s) with 15261 mutant(s). Run estreito (error.ts) matou/sobreviveu mutantes reais (61%, 35 survived).check:depsOK (entradas aceitas) ·check:lockfileOK · YAML válido · semsecretsemif:de job.Status honesto
O nightly produz scores reais para os 8 módulos. Como a fase de mutação completa (15k+ mutantes) excede o dry-run de 20min, o job é advisory com cap de 120min → produz report parcial, não falha. A catraca de
mutationScore.<módulo>fica para a fase INT (depois do 1º nightly calibrar). Não gera versão.