feat(registry): verifier 执照 schema+注册校验（W5-C3 .github#226，ADR-0072） - #82
Merged
Conversation
registry/schemas/verifier-license.json（执照声明契约：考试成绩引用/标注负债 申报/shadow 起步必填）+ scripts/verifier-license.py（考试通过才有条目——对照 CI-Workflows 成绩存档核验；replay 成绩不可注册）+ registry/verifiers/ 执照面 （首版无条目）+ tests 10 例正负向 + golden 快照同步。
|
Warning Review limit reached
Next review available in: 6 minutes Limit details: You’ve used all 10 included reviews currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Comment |
PR Summary by QodoRegistry: add verifier license schema and exam-backed registration validator
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
There was a problem hiding this comment.
Pull request overview
This PR introduces a “verifier license” contract and enforcement point so that new registry/verifiers/*.yaml entries can only be registered when they can be mechanically reconciled against archived CI exam results (ADR-0072 / W5-C3 governance requirement).
Changes:
- Add
registry/schemas/verifier-license.jsonto define the license declaration contract (exam result reference + annotation budget + shadow enforcement). - Add
scripts/verifier-license.pyto validate license entries by cross-checking them against a JSONL exam-results archive (including a--self-testmode). - Add regression tests + golden snapshot update, plus an initial README for
registry/verifiers/.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/verifier-license.py |
New CLI validator that enforces “pass exam before registration” by reconciling YAML entries against JSONL result records. |
registry/schemas/verifier-license.json |
New JSON Schema defining the verifier license declaration shape and required fields. |
registry/verifiers/README.md |
Documentation for the new verifier license registry directory and how to validate entries. |
tests/test_verifier_license.py |
Test suite covering positive/negative validation cases and CLI fail-closed behavior. |
tests/golden/declarations.json |
Golden snapshot updated to include the newly added schema declaration. |
Suppressed comments (1)
scripts/verifier-license.py:77
- 成绩对账目前未核验
model_alias，这会允许用同一 judge_id 的其它模型成绩存档来注册执照条目（条目自报的 model_alias 与成绩不一致也不会被拒）。建议补充对rec.model_alias与entry.model_alias的逐字段校验。
if rec.get("judge_id") != entry.get("judge_id"):
fail(f"{eid}: 成绩 judge_id 不符（{rec.get('judge_id')}）")
if rec.get("exam_version") != exam.get("exam_version"):
fail(f"{eid}: 成绩 exam_version 不符（{rec.get('exam_version')}）")
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+147
to
+150
| for ep in entries: | ||
| entry = yaml.safe_load(Path(ep).read_text(encoding="utf-8")) | ||
| validate_entry(entry, results) | ||
| print(f"校验 {ep}: {'OK' if not errors else 'REJECTED'}") |
Comment on lines
+12
to
+16
| "issued_at", | ||
| "exam", | ||
| "annotation_budget", | ||
| "enforcement" | ||
| ], |
Comment on lines
+65
to
+68
| exam = entry.get("exam") or {} | ||
| # ---- 核心规则：考试通过才有条目（成绩存档对账）---- | ||
| key = exam.get("archive_key") or entry.get("license_id") | ||
| rec = results.get(key) |
Code Review by Qodo
1. Schema misses enforced rules
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+58
to
+68
| m = KEY_RE.match(entry.get("license_id", "")) | ||
| if not m: | ||
| fail(f"{eid}: license_id 须为 judge_id@exam_version@prompt_hash12 形式") | ||
| return | ||
| if m.group("judge") != entry.get("judge_id"): | ||
| fail(f"{eid}: license_id 的 judge 侧与 judge_id 不一致") | ||
|
|
||
| exam = entry.get("exam") or {} | ||
| # ---- 核心规则：考试通过才有条目（成绩存档对账）---- | ||
| key = exam.get("archive_key") or entry.get("license_id") | ||
| rec = results.get(key) |
There was a problem hiding this comment.
1. License key mismatch allowed 🐞 Bug ≡ Correctness
scripts/verifier-license.py does not verify that license_id’s exam_version/prompt_hash match exam.archive_key / the archived exam record; it only checks the judge_id portion, so inconsistent license_id values can be registered while still “passing” against a different archive_key.
Agent Prompt
### Issue description
`scripts/verifier-license.py` validates `license_id` format but only cross-checks the `judge` part against `judge_id`. It then looks up results by `exam.archive_key` (falling back to `license_id`) without enforcing:
- `exam.archive_key == license_id`
- `license_id`’s `exam_version` equals `exam.exam_version`
- `license_id`’s `prompt_hash12` equals `exam.prompt_hash[:12]` and the archived record’s `prompt_hash[:12]`
This allows a YAML entry to present a misleading/incorrect `license_id` while still passing validation by pointing `exam.archive_key` at a different archived record.
### Issue Context
The schema text describes `license_id` must match the archive key (`judge_id@exam_version@prompt_hash12`), so the validator should enforce that as a core integrity rule.
### Fix Focus Areas
- scripts/verifier-license.py[53-83]
- registry/schemas/verifier-license.json[8-50]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+31
to
+33
| REQUIRED_TOP = ("license_id", "judge_id", "model_alias", "issued_at", "exam", | ||
| "annotation_budget", "enforcement") | ||
| HEX64 = re.compile(r"^[0-9a-f]{64}$") |
There was a problem hiding this comment.
2. Model alias not verified 🐞 Bug ⛨ Security
scripts/verifier-license.py requires model_alias on the entry but never checks it against the archived exam record, allowing a license entry to claim a different model than the one that actually passed the exam.
Agent Prompt
### Issue description
The validator treats `model_alias` as required, but it never compares `entry["model_alias"]` to `rec["model_alias"]` from the archived exam results. This is an integrity gap: a registrant can misrepresent which model passed the exam.
### Issue Context
The script already compares multiple fields from the archived record (`judge_id`, `exam_version`, `prompt_hash`, `frozen_exam_sha256`, `judge_mode`, `overall_pass`). `model_alias` should be part of this “逐字段核验” set.
### Fix Focus Areas
- scripts/verifier-license.py[31-33]
- scripts/verifier-license.py[72-86]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+8
to
+16
| "required": [ | ||
| "license_id", | ||
| "judge_id", | ||
| "model_alias", | ||
| "issued_at", | ||
| "exam", | ||
| "annotation_budget", | ||
| "enforcement" | ||
| ], |
There was a problem hiding this comment.
3. Schema misses enforced rules 🐞 Bug ≡ Correctness
registry/schemas/verifier-license.json does not require rubric (but the validator rejects missing rubric), and it does not constrain enforcement.veto to false even though the validator requires shadow start.
Agent Prompt
### Issue description
The JSON schema and the registrar validator disagree:
- The schema’s top-level `required` list omits `rubric`, but `scripts/verifier-license.py` treats rubric fields as mandatory and rejects entries without them.
- The schema allows any boolean for `enforcement.veto`, but the validator requires `false` for new licenses.
This contract drift means schema-only validation and registrar validation can disagree, causing confusing failures and weakening the schema as a normative contract.
### Issue Context
The PR description calls out rubric.annotation_debt and shadow discipline as required semantics. The schema already uses `const` for `exam.overall_pass`, so it can encode `enforcement.veto` similarly.
### Fix Focus Areas
- registry/schemas/verifier-license.json[8-16]
- registry/schemas/verifier-license.json[109-149]
- scripts/verifier-license.py[102-115]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+45
to
+50
| for line in path.read_text(encoding="utf-8").splitlines(): | ||
| if not line.strip(): | ||
| continue | ||
| rec = json.loads(line) | ||
| out[rec["archive_key"]] = rec # 同键多行=多次考试，最新覆盖（历史在行序里） | ||
| return out |
There was a problem hiding this comment.
4. Uncaught parse crashes 🐞 Bug ☼ Reliability
scripts/verifier-license.py can crash on malformed JSONL results or malformed/empty YAML entries (e.g., json.loads errors, missing archive_key, or yaml.safe_load returning None), producing stack traces instead of structured fail-closed errors.
Agent Prompt
### Issue description
The script assumes inputs are well-formed:
- `load_results()` calls `json.loads()` and then `rec["archive_key"]` without exception handling.
- `main()` passes the result of `yaml.safe_load()` directly into `validate_entry()`, which assumes a dict (`entry.get(...)`). Empty YAML yields `None` and will raise `AttributeError`.
This can crash the validator instead of emitting clear `::error::...` messages.
### Issue Context
The script is used as a gate-like validator (“fail-closed”); crashing is still non-zero but makes diagnosis harder and can mask multiple errors.
### Fix Focus Areas
- scripts/verifier-license.py[42-50]
- scripts/verifier-license.py[147-155]
- scripts/verifier-license.py[53-61]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
动机
宪法 §4C 持证上岗 / ADR-0072 决策 2/6：考试通过才有 registry 条目——执照面=
registry 的 verifier 声明结构 + 注册校验脚本。配套 PR：CI-Workflows
w5c3-verifier-exam（考试 CI 与成绩存档）、archive
w5c3-evalsets（考试集冻结正本）。变更清单（全部 append，不动既有条目）
registry/schemas/verifier-license.json：执照声明契约（新增 schema）。必填：成绩引用块（archive_key/exam_version/prompt_hash/frozen_exam_sha256/overall_pass/judge_mode=api）、
标注负债申报块（annual_hours≥1+committed+covers）、enforcement（veto 必须 false=shadow 起步）。
scripts/verifier-license.py：注册校验器（C1 面）——条目必须对上 CI-Workflows 成绩存档JSONL（键=judge_id@exam_version@prompt_hash12，逐字段核验）；replay 回放成绩不可注册；
未配标注预算/新发 veto=true 一律拒绝；
--self-test内置 10 组正负向断言。registry/verifiers/README.md：执照面说明（首版无条目——尚无真实判官通过 api 模式考试）。tests/test_verifier_license.py：10 例（正例放行 + 逐项缺陷注入必拒 + CLI fail-closed）。tests/golden/declarations.json：快照同步（新增 1 个声明文件的语义面）。AC 映射（.github#226）
agent-registry——本 PR 是"才注册"的执行点：
verifier-license.py对照成绩存档核验（
test_missing_archive_rejected：无成绩=拒；test_failed_exam_rejected：分项不过=拒；test_replay_result_rejected：回放成绩=拒）。rubric.annotation_debt（insufficient-data+reason 结构化），未配预算不许上岗（§10.4）。
测试方法（本地已跑，零网络）
python -m pytest tests/test_verifier_license.py -v→ 10 例全绿；python scripts/verifier-license.py --self-test→ 10 组 PASS。scripts/validate.pyOK（tools=5 skills=2 agents=9 teams=3 models=5）；scripts/snapshot.py --checkOK（62 个声明文件）；python -m pytest tests/70 passed；scripts/simulate-wave.pyexit 0。风险与回滚
（README 有操作说明）；长期存档转仓库为 follow-up。
registry 既有条目零改动。
Card: Cloudbird-Software/.github#226
ADR: ADR-0072（archive
adr/ADR-0072-verifier-entrance-exam-calibration.md）