Resolve binlog MCP from the dnceng dotnet-public feed

[YuliiaKovalova]

Summary

The binlog MCP server was declared as dnx Microsoft.AITools.BinlogMcp --yes with no feed pinned, so Microsoft.AITools.BinlogMcp resolved through whatever ambient NuGet configuration the agent happened to run under (user-level NuGet.Config, an enclosing repo's NuGet.config, etc.). That makes startup environment-dependent and hard to reproduce.

This pins the server to the dnceng dotnet-public feed via --source:

"args": [
  "dnx",
  "Microsoft.AITools.BinlogMcp",
  "--yes",
  "--source",
  "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json"
]

--source (rather than --add-source) replaces all sources, so resolution is fully deterministic. This is safe here because .NET tool packages bundle their dependencies — only the single package needs to be restored.

The existing argument order is preserved, so the change is a pure insertion with no reordering.

Files changed

The server definition is duplicated across four manifests, all updated together so the mirrors stay in sync:

• plugins/dotnet-msbuild/plugin.json
• plugins/dotnet-msbuild/.claude-plugin/plugin.json
• plugins/dotnet-msbuild/.codex-plugin/.mcp.json
• .claude-plugin/marketplace.json

Two stale feed names are also corrected:

• eng/allowed-external-deps.txt — comment said dotnet-eng feed
• plugins/dotnet-msbuild/skills/binlog-failure-analysis/SKILL.md — fallback note said dotnet-tools NuGet feed

plugins/dotnet-msbuild/.codex-plugin/plugin.json needs no change — it delegates via "mcpServers": "./.mcp.json".

Validation

• Package availability confirmed on dotnet-public: versions 1.0.0, 2.0.0, 2.0.1 (all stable, so --prerelease is not needed).

• End-to-end server start against the pinned feed, piping an MCP initialize request into the server:

  Server (aitools-binlog-mcp 2.0.1) method 'initialize' request handler called.
  Server (aitools-binlog-mcp 2.0.1), Client (probe 1.0) method 'initialize' request handler completed in 20.5649ms.
  

• CI gate reproduced locally — skill-validator check over all 16 plugins with --allowed-external-deps and --known-domains exits 0: 96 skills, 16 agents, 208 files scanned, 0 reference errors and no new warnings.

• No known-domains.txt entry is required: ReferenceScanner only scans SKILL.md, *.agent.md, and references/*.md — not plugin.json.

Note on the failing eval-quality check

The check / Check eval quality job failure is pre-existing on main and unrelated to this PR — it fails on 4 stale tests/dotnet-maui/* entries in eng/eval-quality/underpowered-allowlist.txt that now meet the 5-trial floor:

eng/eval-quality/underpowered-allowlist.txt lists 'tests/dotnet-maui/maui-collectionview/eval.yaml',
but it now meets the 5-trial floor. Remove the line so the exemption can't be silently reused.

Same failure on main: run 30532730249. This PR touches no files under tests/ or eng/eval-quality/.

Related

Companion PR: dotnet/arcade-skills#49 makes the same change for the dotnet-dnceng plugin, which publishes a binlog server under the same name.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-msbuild	check-bin-obj-clash	4/5	80%
❌	dotnet-msbuild	directory-build-organization	0/1	0%
❌	dotnet-msbuild	extension-points	0/1	0%
✅	dotnet-msbuild	msbuild-modernization	6/7	85.7%
⚠️	dotnet-msbuild	msbuild-server	7/9	77.8%
❌	dotnet-msbuild	property-patterns	0/1	0%
✅	dotnet-msbuild	resolve-project-references	5/6	83.3%

Uncovered: dotnet-msbuild/check-bin-obj-clash

• [WorkflowStep] Step 2: Get an overview and list projects (line 48)

Uncovered: dotnet-msbuild/directory-build-organization

• [CodePattern] [MSBuild] (line 122)

Uncovered: dotnet-msbuild/extension-points

• [CodePattern] [MSBuild] (line 195)

Uncovered: dotnet-msbuild/msbuild-modernization

• [WorkflowStep] Step 6: Remove Unnecessary Boilerplate (line 217)

Uncovered: dotnet-msbuild/msbuild-server

• [Validation] MSBUILDUSESERVER=1 is set in the shell (line 59)
• [Validation] Second sequential build is faster than the first (line 60)

Uncovered: dotnet-msbuild/property-patterns

• [CodePattern] [MSBuild] (line 33)

Uncovered: dotnet-msbuild/resolve-project-references

• [Validation] ResolveProjectReferences was not set as the optimization target (line 64)

[Copilot]

Pull request overview

This PR aims to make the binlog MCP server startup deterministic by pinning Microsoft.AITools.BinlogMcp restore to the dnceng dotnet-public NuGet feed, avoiding dependence on ambient NuGet configuration on the host environment.

Changes:

• Updated the binlog MCP server invocation to include a pinned NuGet source (--source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json) across the mirrored manifests.
• Corrected stale feed wording in documentation and the allowed-external-deps comment to reference dotnet-public.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
plugins/dotnet-msbuild/skills/binlog-failure-analysis/SKILL.md	Updates fallback documentation to reference the correct feed name (dotnet-public).
plugins/dotnet-msbuild/plugin.json	Adds a pinned NuGet feed to the binlog MCP server definition.
plugins/dotnet-msbuild/.codex-plugin/.mcp.json	Mirrors the pinned-feed binlog MCP server definition for Codex plugin consumption.
plugins/dotnet-msbuild/.claude-plugin/plugin.json	Mirrors the pinned-feed binlog MCP server definition for Claude plugin consumption.
eng/allowed-external-deps.txt	Updates the explanatory comment to reflect the dotnet-public feed.
.claude-plugin/marketplace.json	Mirrors the pinned-feed binlog MCP server definition in the marketplace manifest.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[YuliiaKovalova]

/evaluate d70cc01

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	+100.0%	0.500	+100.0%	1/0/0	5.0/5	0.0/5	✅ 0.06	1/1 · 1/1 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +100.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Diagnose build failures from binlog only (no source files)	+100.0%	+100.0%	1/0/0

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[ViktorHofer]

This looks questionable to me. We should never promote a package to customers from the dotnet-public nuget feed. That's a public feed but exists only for our own engineering infrastructure (to build .NET). cc @mmitche to double check my statement

[github-actions]

👋 @YuliiaKovalova — this PR has changes requested. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[YuliiaKovalova]

This looks questionable to me. We should never promote a package to customers from the dotnet-public nuget feed. That's a public feed but exists only for our own engineering infrastructure (to build .NET). cc @mmitche to double check my statement

I had conversation about feeds here befroe dotnet/dotnet-buildtools-prereqs-docker#1682 (comment)

these cases look similar to me, but i might be wrong.

[github-actions]

👋 @YuliiaKovalova — this PR has changes requested. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Evangelink]

/evaluate

[github-actions]

👋 /evaluate needs the exact commit to evaluate.

Two ways to run it:

1. Review flow (recommended — no SHA to copy): open Files changed → Review changes, type /evaluate in the review box, and Submit review. GitHub binds the run to the exact commit you reviewed.
2. Comment flow: copy-paste this command for the current head:

/evaluate 73e8ad89848d842de30bccc9aec1e0b5dd71419d

[Copilot]

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 0 new

got confirmation that change is correct

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	+0.0%	1.000	+0.0%	0/1/0	0.0/5	0.0/5	✅ 0.08	1/1 · 1/1 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose build failures from binlog only (no source files)	+0.0%	+0.0%	0/1/0

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[ViktorHofer]

got confirmation that change is correct

Apparently requesting changes doesn't matter anymore? Next time share context before dismissing a change request or even better, don't dismiss at all. That doesn't feel right.

[YuliiaKovalova]

got confirmation that change is correct

Apparently requesting changes doesn't matter anymore? Next time share context before dismissing a change request or even better, don't dismiss at all. That doesn't feel right.

I apologize, it looked to me like I had enough evidence to proceed.
If you have technical concerns I can revert this change and start the discussion from scratch.

[ViktorHofer]

Thanks for being considerate, really appreciate that. Let's move the discussion offline. I opened a Teams channel to collaborate further on the dotnet-msbuild plugin.

[JanKrivanek]

@YuliiaKovalova - were there any cases where the tooling didn't work as expected?

I'd prefer leaving the feed definition up to destination env - similar to e.g. Roslyn lsp https://github.com/dotnet/skills/blob/main/plugins/dotnet/lsp.json