Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/scheduled.yml
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,13 @@ jobs:
# and the registry's verbatim quotes, both re-verified weekly. Failures
# here page the schedule, not PRs — an expired deferral or a drifted quote
# is a maintenance event, not a defect in unrelated work.
#
# 2026-08-18: that split is still right, but "pages the schedule" currently
# means GitHub's default scheduled-failure email and nothing else, and two
# rows expired unnoticed behind it. Making the failure durable (a tracking
# issue naming the expired rows) needs `issues: write` on a job that is
# deliberately `contents: read`, so it is its own change — deferral
# `expired-deferral-notification`, review-by 2026-11-01.
claims-expire:
name: Deferral ledger + spec-quote drift (ADR-0010)
runs-on: ubuntu-latest
Expand Down
87 changes: 87 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,55 @@ Pre-1.0, minor releases may contain breaking changes; entries say so explicitly.
could not fix. The task now passes `--all-features`, so all three `draft::`
golden tests run.

- **Two deferral rows sat expired, and three published claims had rotted behind
them.** `cargo xtask deferrals --check` is a weekly scheduled gate, not a PR
gate (ADR-0010, deliberately — an expiry should not block unrelated work), and
nobody acted on it: `rust-sdk-902-offer-clock` passed review-by on 2026-08-10
and `suite-0-2-0-stable-pin-bump` on 2026-08-15. Both are re-decided against
re-fetched evidence rather than re-dated blind, and the corrected claims are:

- The ledger and [register row 2.4](docs/plan/01-ecosystem-context.md) said the
npm `alpha` dist-tag "has been quiet since 2026-07-01". It was not: `alpha.10`
published **2026-07-27, the day after that row was written**. This is
[ADR-0010](docs/plan/decisions/0010-deferral-ledger-and-scheduled-reverification.md)'s
own founding example recurring verbatim, so the row now states its
observation window instead of predicting upstream quiet.
- [07-ecosystem-engagement.md](docs/plan/07-ecosystem-engagement.md) called the
`enumNames` fix ([rust-sdk#905](https://github.com/modelcontextprotocol/rust-sdk/pull/905))
"maintainer-authored". GitHub shows the author carrying the **Contributor**
badge, approved and merged by a **Member**. The engagement was still
successful; the distinction is precisely the one risk R9 measures.
- **R9 has not fired.** Its trigger is *two* substantive offers unanswered for
60+ days. [rust-sdk#902](https://github.com/modelcontextprotocol/rust-sdk/issues/902)
is unanswered at 68 days — open, zero comments, no assignee, no linked PR —
but the same day's [#903](https://github.com/modelcontextprotocol/rust-sdk/issues/903)
was answered and fixed in nine days. The count stands at one, so M4's DoD
does not re-scope. Recorded in the risk register with the evidence.

- **The vacuous-pass arithmetic survived in the plan docs, because the gate that
swept it does not reach them.** The 2026-08-17 correction fixed "123 and 124
passes where the reports say 58 pass, 1 fail and 59 pass, 0 fail" in the
CLAIM_FILES `cargo xtask draft-coverage --check` parses. It stopped exactly
there: [register row 1.5i](docs/plan/01-ecosystem-context.md),
[03-conformance-strategy.md](docs/plan/03-conformance-strategy.md) and
[06-roadmap.md](docs/plan/06-roadmap.md) are outside that set and still carried
the inflated pair — `pass + not-observed`, the very accounting
[ADR-0012](docs/plan/decisions/0012-not-observed-outcome.md) removed. All three
corrected, with the register row recording what it used to say and why.

It was found the right way: a CHANGELOG entry here quoted register row 1.5i,
and the claim gate rejected the verdict as one no committed report produced.
The lesson is the gate's boundary, not the arithmetic — a hand-kept number
outside the checked set drifts silently, and the checked set is currently seven
Markdown files.

Two new rows open for what this exposed: `draft-suite-pin-currency` (the
ratchet's input is pre-release and needs dated re-checking, since the weekly
alpha job runs at the *registry's* revision and cannot see draft scenario
churn) and `expired-deferral-notification` (a red weekly job currently reaches
a human only by an easily-missed email; making it durable needs `issues: write`
on a `contents: read` workflow, so it gets its own reviewed change).


- **The reference server answered a missing resource with a code
`2026-07-28` withdrew.** `resources/read` for a URI it does not serve drew
Expand Down Expand Up @@ -521,6 +570,44 @@ Pre-1.0, minor releases may contain breaking changes; entries say so explicitly.
6.55 MB become 1.32 MB. See
[ADR-0013](docs/plan/decisions/0013-golden-report-format.md).

- **The draft-readiness ratchet moves to suite `0.2.0-alpha.11`, and the runner
can now tell the two servers apart — agreeing with this workspace's registry
on the clause it found first.** `DRAFT_SUITE_VERSION` had sat on
`0.2.0-alpha.9` (2026-07-01) for six weeks while `alpha.10` (2026-07-27) and
`alpha.11` (2026-08-07) shipped. Re-measured with `BLESS=1`: **no pre-existing
check changed status**, and the entire delta is 36 new `wire-schema-valid`
checks, which validate every message against the negotiated revision's JSON
schema. Thirty-two pass. The four that fail are all on the `2025-11-25` leg —
`resources-{list,read-text,read-binary,templates-read}`, each for `must have
required property 'cacheScope'` and `'ttlMs'`.

That is **CACH-001**, the single clause the registry here had already flagged
against the legacy server — the two captures read 58 pass, 1 fail and
59 pass, 0 fail, with 65 clauses not observed on each — while the official
runner scored both servers an indistinguishable 23/23. The runner has now found it
independently, six weeks later. The standing finding "the runner cannot
distinguish the two servers" is superseded rather than deleted, in
[register row 1.5i](docs/plan/01-ecosystem-context.md) and
[06-roadmap.md](docs/plan/06-roadmap.md): a negative result about an
instrument expires when the instrument improves. Legs now score 37 passing /
4 failing (legacy) and 41 / 0 (stateless).

The asymmetry is instructive and is recorded: `tools/list` and `prompts/list`
pass because rmcp's `#[tool_handler]`/`#[prompt_handler]` expansions attach
caching hints unconditionally, while `resources/*` go through this workspace's
revision-aware `cached()`, which correctly withholds them at `2025-11-25`. The
honest implementation is the one the new check fails.

- **The two suite pins are no longer coupled.**
[03-conformance-strategy.md](docs/plan/03-conformance-strategy.md) said both
move when the `0.2.0` line stabilizes. They have different triggers:
`SUITE_VERSION` gates the *released* revision and waits for a stable release
to exist (`0.1.16`, unchanged since 2026-03-30, is still the only one);
`DRAFT_SUITE_VERSION` measures readiness against a scenario set that is itself
pre-release and moving, so holding it back does not keep the measurement
stable — it makes it describe an older question. Six weeks on `alpha.9` cost
exactly that.

- **Breaking (pre-1.0):** `mcp_everything_server::http::router` and
`http::router_tapped` take a `ServedRevision`. Pass
`ServedRevision::default()` (or `ServedRevision::V2025_11_25`) for the
Expand Down
114 changes: 75 additions & 39 deletions conformance/draft-readiness.json
Original file line number Diff line number Diff line change
@@ -1,136 +1,172 @@
{
"_policy": "Measured by `cargo xtask draft-readiness`: the status of every check the official runner's 2026-07-28 scenarios report against the everything server, once per revision the server can serve. Scenario keys are `<served revision>/<scenario>`: the 2025-11-25 leg measures migration distance, the 2026-07-28 leg measures the server's own stateless mode. The gate fails on ANY change — a lost pass is a regression, a gained one is progress that gets recorded deliberately (BLESS=1) in the commit that earned it. Statuses are verbatim: INFO is the runner's informational outcome and is neither a pass nor a failure. Passing the runner is not the same as conforming: these scenarios come from a pre-release suite, so the check set itself can still move, and the runner judges a subset of what the registry in this workspace judges.",
"failing": 0,
"failing": 4,
"informational": 0,
"passing": 46,
"passing": 78,
"scenarios": {
"2025-11-25/completion-complete": {
"completion-complete": "SUCCESS"
"completion-complete": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/dns-rebinding-protection": {
"localhost-host-rebinding-rejected": "SUCCESS",
"localhost-host-valid-accepted": "SUCCESS"
},
"2025-11-25/prompts-get-embedded-resource": {
"prompts-get-embedded-resource": "SUCCESS"
"prompts-get-embedded-resource": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/prompts-get-simple": {
"prompts-get-simple": "SUCCESS"
"prompts-get-simple": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/prompts-get-with-args": {
"prompts-get-with-args": "SUCCESS"
"prompts-get-with-args": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/prompts-get-with-image": {
"prompts-get-with-image": "SUCCESS"
"prompts-get-with-image": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/prompts-list": {
"prompts-list": "SUCCESS"
"prompts-list": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/resources-list": {
"resources-list": "SUCCESS"
"resources-list": "SUCCESS",
"wire-schema-valid": "FAILURE"
},
"2025-11-25/resources-read-binary": {
"resources-read-binary": "SUCCESS"
"resources-read-binary": "SUCCESS",
"wire-schema-valid": "FAILURE"
},
"2025-11-25/resources-read-text": {
"resources-read-text": "SUCCESS"
"resources-read-text": "SUCCESS",
"wire-schema-valid": "FAILURE"
},
"2025-11-25/resources-templates-read": {
"resources-templates-read": "SUCCESS"
"resources-templates-read": "SUCCESS",
"wire-schema-valid": "FAILURE"
},
"2025-11-25/server-sse-multiple-streams": {
"server-accepts-multiple-post-streams": "SUCCESS",
"server-sse-streams-functional": "SUCCESS"
},
"2025-11-25/tools-call-audio": {
"tools-call-audio": "SUCCESS"
"tools-call-audio": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-call-embedded-resource": {
"tools-call-embedded-resource": "SUCCESS"
"tools-call-embedded-resource": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-call-error": {
"tools-call-error": "SUCCESS"
"tools-call-error": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-call-image": {
"tools-call-image": "SUCCESS"
"tools-call-image": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-call-mixed-content": {
"tools-call-mixed-content": "SUCCESS"
"tools-call-mixed-content": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-call-simple-text": {
"tools-call-simple-text": "SUCCESS"
"tools-call-simple-text": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-call-with-progress": {
"tools-call-with-progress": "SUCCESS"
"tools-call-with-progress": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2025-11-25/tools-list": {
"tools-list": "SUCCESS",
"tools-name-format": "SUCCESS"
"tools-name-format": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/completion-complete": {
"completion-complete": "SUCCESS"
"completion-complete": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/dns-rebinding-protection": {
"localhost-host-rebinding-rejected": "SUCCESS",
"localhost-host-valid-accepted": "SUCCESS"
},
"2026-07-28/prompts-get-embedded-resource": {
"prompts-get-embedded-resource": "SUCCESS"
"prompts-get-embedded-resource": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/prompts-get-simple": {
"prompts-get-simple": "SUCCESS"
"prompts-get-simple": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/prompts-get-with-args": {
"prompts-get-with-args": "SUCCESS"
"prompts-get-with-args": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/prompts-get-with-image": {
"prompts-get-with-image": "SUCCESS"
"prompts-get-with-image": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/prompts-list": {
"prompts-list": "SUCCESS"
"prompts-list": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/resources-list": {
"resources-list": "SUCCESS"
"resources-list": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/resources-read-binary": {
"resources-read-binary": "SUCCESS"
"resources-read-binary": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/resources-read-text": {
"resources-read-text": "SUCCESS"
"resources-read-text": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/resources-templates-read": {
"resources-templates-read": "SUCCESS"
"resources-templates-read": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/server-sse-multiple-streams": {
"server-accepts-multiple-post-streams": "SUCCESS",
"server-sse-streams-functional": "SUCCESS"
},
"2026-07-28/tools-call-audio": {
"tools-call-audio": "SUCCESS"
"tools-call-audio": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-call-embedded-resource": {
"tools-call-embedded-resource": "SUCCESS"
"tools-call-embedded-resource": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-call-error": {
"tools-call-error": "SUCCESS"
"tools-call-error": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-call-image": {
"tools-call-image": "SUCCESS"
"tools-call-image": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-call-mixed-content": {
"tools-call-mixed-content": "SUCCESS"
"tools-call-mixed-content": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-call-simple-text": {
"tools-call-simple-text": "SUCCESS"
"tools-call-simple-text": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-call-with-progress": {
"tools-call-with-progress": "SUCCESS"
"tools-call-with-progress": "SUCCESS",
"wire-schema-valid": "SUCCESS"
},
"2026-07-28/tools-list": {
"tools-list": "SUCCESS",
"tools-name-format": "SUCCESS"
"tools-name-format": "SUCCESS",
"wire-schema-valid": "SUCCESS"
}
},
"spec": "2026-07-28",
"suite": "0.2.0-alpha.9"
"suite": "0.2.0-alpha.11"
}
30 changes: 21 additions & 9 deletions corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,12 +173,12 @@ two reports is attributable to that one change.

| Field | `official-suite-2026-07-28-scenarios.jsonl` | `official-suite-2026-07-28-stateless.jsonl` |
|---|---|---|
| Client | The **official MCP conformance suite**, `0.2.0-alpha.9`, driving its `2026-07-28` scenario set (the pin `cargo xtask draft-readiness` holds) | The same client, the same scenarios, the same run |
| Client | The **official MCP conformance suite**, `0.2.0-alpha.9`, driving its `2026-07-28` scenario set (the pin `cargo xtask draft-readiness` held when these were recorded; it moved to `0.2.0-alpha.11` on 2026-08-18, and re-recording is a deliberate act — see below) | The same client, the same scenarios, the same run |
| Server | `mcp-everything-server` serving **`2025-11-25`** — held to a revision it does not implement, so genuine non-conformance is the expected content | `mcp-everything-server --protocol-version 2026-07-28`, its stateless mode |
| Recorded by | `mcp-everything-server`'s tap, during `cargo xtask draft-readiness`, 2026-08-17 | same run, second leg |
| Contents | 91 events / 22 POST exchanges | 91 events / 22 POST exchanges |
| Our verdict | 58 pass, **1 fail**, 0 warn, 65 not observed, 148 excluded | **59 pass, 0 fail, 0 warn**, 65 not observed, 148 excluded |
| The official runner's verdict | 23/23 | 23/23 |
| The official runner's verdict | 23/23 at `alpha.9`; **37 passing / 4 failing** at `alpha.11` | 23/23 at `alpha.9`; **41 passing / 0 failing** at `alpha.11` |

Both carry `server/discover`, `tools/list`, `tools/call`, `completion/complete`,
`resources/{list,read}`, `prompts/{list,get}` and progress notifications; every
Expand Down Expand Up @@ -207,13 +207,25 @@ than quietly fixed: **a check can only be as honest as the recording it reads,
and a capture path that silently drops evidence manufactures findings against
conforming implementations.**

**Why the runner's 23/23 and our 58-vs-59 are both right.** The suite's
`2026-07-28` scenarios exercise features — list a thing, call a thing, read a
thing — and a `2025-11-25` server answers all of them, because rmcp serves a
per-request-versioned POST whichever revision the handler advertises. The
registry here judges the specification's prose instead, so it sees the one place
the two servers actually differ. Neither instrument is wrong; they are measuring
different things, and this pair is the evidence for that.
**Why the runner's 23/23 and our 58-vs-59 were both right — and how the runner
caught up.** The suite's `2026-07-28` scenarios exercise features — list a
thing, call a thing, read a thing — and a `2025-11-25` server answers all of
them, because rmcp serves a per-request-versioned POST whichever revision the
handler advertises. At `0.2.0-alpha.9` that was the whole of the runner's
reading, so it scored both servers 23/23. The registry here judges the
specification's prose instead, so it saw the one place the two servers actually
differ: CACH-001, no `ttlMs` on cacheable results.

**`0.2.0-alpha.11` found the same thing, six weeks later.** Its new
`wire-schema-valid` check validates every message against the negotiated
revision's JSON schema, and it fails the `2025-11-25` leg's four resource
scenarios for `must have required property 'cacheScope'` / `'ttlMs'` — that
clause, independently. Re-measured, the legacy leg scores 37 passing / 4
failing and the stateless leg 41 / 0, so the runner now separates the two
servers it could not tell apart before. Neither instrument was wrong; they were
measuring different things, and one of them saw this first. The pair is still
the evidence for taking both readings — now with a worked example of the
prose-level reading arriving earlier than the schema-level one.

The 65 not-observed rows are the honest denominator: of the 124 clauses this
revision's registry can judge, these sessions carried subject matter for 59.
Expand Down
4 changes: 3 additions & 1 deletion crates/mcp-everything-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@ server scenarios (pinned suite 0.1.16, enforced in CI via
caching hints on cacheable results. An `initialize` sent to it is refused
with `-32022` naming the versions it does speak, rather than negotiated into
a handshake that leads nowhere. The suite's `2026-07-28` scenarios pass
**23/23** against this mode, and the repository's own requirement registry
**41/41** against this mode at the pinned `0.2.0-alpha.11` (23/23 at the
earlier `alpha.9` pin, before its `wire-schema-valid` check existed), and the
repository's own requirement registry
evidences **109 of the 124 judgeable clauses** across five committed captures
of it over both transports, with 0 fail on the conforming ones and everything
the sessions never touched reported *not observed* rather than counted as a
Expand Down
Loading