Make BinlogMcp feed-neutral and configure eval NuGet source

[Evangelink]

Reverts #969 while keeping package-source ownership in the evaluation environment.

Summary

• Removes the explicit dotnet-public source from every Microsoft.AITools.BinlogMcp plugin manifest.
• Adds a repository-owned NuGet configuration for evaluation runs that clears ambient sources and maps all packages to dotnet-public.
• Installs that configuration at the runner user level so Vally's temporary workspaces discover it.
• Makes related documentation source-neutral.

Validation

• Installed Microsoft.AITools.BinlogMcp 2.0.1 with --no-cache from dotnet-public using only eng/evaluation/nuget.config.
• Confirmed user-level NuGet discovery from an unrelated temporary work directory.
• Parsed all changed JSON and XML files successfully.
• actionlint and git diff --check pass.
• The full skill-validator publish restored through the explicit NuGet configuration, then was blocked by an external npm TLS handshake failure while downloading @github/copilot-win32-x64.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-msbuild	check-bin-obj-clash	4/5	80%
❌	dotnet-msbuild	directory-build-organization	0/1	0%
❌	dotnet-msbuild	extension-points	0/1	0%
✅	dotnet-msbuild	msbuild-modernization	6/7	85.7%
⚠️	dotnet-msbuild	msbuild-server	7/9	77.8%
❌	dotnet-msbuild	property-patterns	0/1	0%
✅	dotnet-msbuild	resolve-project-references	5/6	83.3%

Uncovered: dotnet-msbuild/check-bin-obj-clash

• [WorkflowStep] Step 2: Get an overview and list projects (line 48)

Uncovered: dotnet-msbuild/directory-build-organization

• [CodePattern] [MSBuild] (line 122)

Uncovered: dotnet-msbuild/extension-points

• [CodePattern] [MSBuild] (line 195)

Uncovered: dotnet-msbuild/msbuild-modernization

• [WorkflowStep] Step 6: Remove Unnecessary Boilerplate (line 217)

Uncovered: dotnet-msbuild/msbuild-server

• [Validation] MSBUILDUSESERVER=1 is set in the shell (line 59)
• [Validation] Second sequential build is faster than the first (line 60)

Uncovered: dotnet-msbuild/property-patterns

• [CodePattern] [MSBuild] (line 33)

Uncovered: dotnet-msbuild/resolve-project-references

• [Validation] ResolveProjectReferences was not set as the optimization target (line 64)

[Copilot]

Pull request overview

This PR reverts PR #969 by removing the pinned NuGet feed (--source ...dotnet-public...) from the Microsoft.AITools.BinlogMcp MCP server declarations, returning package resolution to whatever NuGet sources are configured in the execution environment. It also reverts related documentation text that was updated alongside the feed pinning work.

Changes:

• Removed --source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json from the BinlogMcp dnx arguments across the mirrored manifests.
• Reverted feed-name documentation text in eng/allowed-external-deps.txt and the binlog failure analysis skill.

Show a summary per file

File	Description
plugins/dotnet-msbuild/skills/binlog-failure-analysis/SKILL.md	Reverts the fallback workflow wording about which feed may be unavailable.
plugins/dotnet-msbuild/plugin.json	Removes the pinned --source from the BinlogMcp MCP server args.
plugins/dotnet-msbuild/.codex-plugin/.mcp.json	Removes the pinned --source from the BinlogMcp MCP server args (Codex mirror).
plugins/dotnet-msbuild/.claude-plugin/plugin.json	Removes the pinned --source from the BinlogMcp MCP server args (Claude mirror).
eng/allowed-external-deps.txt	Reverts the comment describing which feed the BinlogMcp server comes from.
.claude-plugin/marketplace.json	Removes the pinned --source from the BinlogMcp MCP server args (marketplace mirror).

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 6/6 changed files
• Comments generated: 2

[github-actions]

👋 @Evangelink — this PR has 2 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[JanKrivanek]

For external customers the primary source feed for Microsoft.AITools.BinlogMcp server should be nuget.org

The question is what it will do in our eval env

[JanKrivanek]

For external customers the primary source feed for Microsoft.AITools.BinlogMcp server should be nuget.org

The question is what it will do in our eval env

We probably want to remove the feeds mentions alltogether and explicitly set the feed for the eval environment.

IoW - the feed should be controlled by the execution env, not by the AI pluggable logic

[Copilot]

Copilot's findings

• Files reviewed: 8/8 changed files
• Comments generated: 0 new

[JanKrivanek]

/evaluate

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	+0.0%	1.000	+0.0%	0/1/0	5.0/5	4.6/5	✅ 0.06	1/1 · 1/1 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose build failures from binlog only (no source files)	+0.0%	+0.0%	0/1/0

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for e83754c. cc @webreidi @AbhitejJohn @JanKrivanek @ViktorHofer @dotnet/skills-msbuild-reviewers — please review.

[Copilot]

Copilot's findings

Suppressed comments (1)

eng/version/nuget.config:26

• eng/version/nuget.config is now pinned to the dotnet-public feed, but the versioning workflows that consume this file still document that tools are restricted to nuget.org (e.g. .github/workflows/version-bump-command.yml says “tools can only be restored from nuget.org”). This mismatch makes it unclear which feed is the intended trust boundary for privileged tool restores, and it may surprise maintainers if dotnet-public is ever unavailable or doesn’t mirror a required tool version.

Please either (a) revert this file back to nuget.org, or (b) update the versioning workflow documentation/comments (and ideally the PR description) to explicitly state that the trusted source is dotnet-public and why.

  <packageSources>
    <clear />
    <add key="dotnet-public" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="dotnet-public">
      <package pattern="*" />

• Files reviewed: 9/9 changed files
• Comments generated: 0 new

[JanKrivanek]

/evaluate

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	-100.0%	0.500	-100.0%	0/0/1	0.0/5	4.6/5	✅ 0.08	1/1 · 1/1 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win -100.0% (0W/0T/1L over 1 trial(s), sign test p=0.500), mean preference -100.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
▼ Diagnose build failures from binlog only (no source files)	-100.0%	-100.0%	0/0/1

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Copilot]

Copilot's findings

Suppressed comments (5)

plugins/dotnet-msbuild/plugin.json:19

• --prerelease makes dotnet dnx Microsoft.AITools.BinlogMcp eligible to resolve to newer prerelease builds, which can introduce unexpected behavior changes and reduce reproducibility. Unless this server package is only available as prerelease, prefer restoring the latest stable (or pin an explicit version) and drop --prerelease.

        "dnx",
        "Microsoft.AITools.BinlogMcp",
        "--yes",
        "--prerelease"

plugins/dotnet-msbuild/.codex-plugin/.mcp.json:10

• --prerelease makes dotnet dnx Microsoft.AITools.BinlogMcp eligible to resolve to newer prerelease builds, which can introduce unexpected behavior changes and reduce reproducibility. Unless this server package is only available as prerelease, prefer restoring the latest stable (or pin an explicit version) and drop --prerelease.

        "dnx",
        "Microsoft.AITools.BinlogMcp",
        "--yes",
        "--prerelease"

plugins/dotnet-msbuild/.claude-plugin/plugin.json:19

• --prerelease makes dotnet dnx Microsoft.AITools.BinlogMcp eligible to resolve to newer prerelease builds, which can introduce unexpected behavior changes and reduce reproducibility. Unless this server package is only available as prerelease, prefer restoring the latest stable (or pin an explicit version) and drop --prerelease.

        "dnx",
        "Microsoft.AITools.BinlogMcp",
        "--yes",
        "--prerelease"

.claude-plugin/marketplace.json:39

• --prerelease makes dotnet dnx Microsoft.AITools.BinlogMcp eligible to resolve to newer prerelease builds, which can introduce unexpected behavior changes and reduce reproducibility. Unless this server package is only available as prerelease, prefer restoring the latest stable (or pin an explicit version) and drop --prerelease.

            "dnx",
            "Microsoft.AITools.BinlogMcp",
            "--yes",
            "--prerelease"

eng/version/nuget.config:25

• This switches the locked-down versioning NuGet config from nuget.org to dotnet-public. That’s a behavior change for the versioning workflows (dotnet tool restore --configfile eng/version/nuget.config) and isn’t called out in the PR description (which focuses on evaluation NuGet config). Please confirm this is intentional; if not, revert this file to keep versioning restores on nuget.org.

    <clear />
    <add key="dotnet-public" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="dotnet-public">

• Files reviewed: 9/9 changed files
• Comments generated: 0 new

[github-actions]

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	+0.0%	1.000	+0.0%	0/1/0	5.0/5	5.0/5	✅ 0.06	1/1 · 1/1 (plugin)

ℹ️ Column legend

• Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
• p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
• Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
• W/T/L — wins / ties / losses across trials.
• ⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
• 🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
• Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
• Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose build failures from binlog only (no source files)	+0.0%	+0.0%	0/1/0

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Approved by @ViktorHofer. cc @dotnet/skills-merge-approvers — ready to merge.