add a new dotnet11 plugin and a new system text json skill to it. - #535
Conversation
|
@sayedihashimi - FYI |
There was a problem hiding this comment.
Pull request overview
Adds a new dotnet11 plugin focused on .NET 11-era guidance and introduces an initial System.Text.Json skill with a corresponding evaluation suite.
Changes:
- Added new
dotnet11plugin scaffolding (plugin manifest + plugin README). - Added
system-text-json-net11skill documentation and a new eval suite undertests/dotnet11/. - Registered the new plugin in the root README, marketplace manifests, and CODEOWNERS; also added
*.lscacheto.gitignore.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet11/system-text-json-net11/eval.yaml | Adds 3 evaluation scenarios for the new System.Text.Json .NET 11 skill (including a non-activation case). |
| plugins/dotnet11/skills/system-text-json-net11/SKILL.md | Documents the new System.Text.Json .NET 11 skill and its APIs/examples. |
| plugins/dotnet11/plugin.json | Declares the new dotnet11 plugin manifest. |
| plugins/dotnet11/README.md | Adds a short plugin-level README listing included skills. |
| README.md | Adds dotnet11 to the “What’s Included” plugin table. |
| .gitignore | Ignores Roslyn/C# language server cache files (*.lscache). |
| .github/plugin/marketplace.json | Registers dotnet11 in the GitHub plugin marketplace manifest. |
| .claude-plugin/marketplace.json | Mirrors the marketplace registration for Claude plugin manifest. |
| .github/CODEOWNERS | Adds ownership entries for the new plugin, skill, and tests. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (4)
tests/dotnet11/system-text-json-net11/eval.yaml:101
- Scenario 3 rubric requires producing camelCase JSON output like
{"name":"Jane","age":30}, but the assertions only check thatJsonNamingPolicy.CamelCaseis mentioned. Add an assertion that matches the actual JSON output (or at least verifies the output contains lowercase"name"/"age") so the scenario can’t pass without demonstrating the correct serialization result.
assertions:
- type: "exit_success"
- type: "output_matches"
pattern: "net8\\.0"
- type: "output_matches"
pattern: "JsonNamingPolicy\\.CamelCase"
- type: "output_not_matches"
pattern: "JsonNamingPolicy\\.PascalCase"
- type: "output_not_matches"
pattern: "net11\\.0"
rubric:
- "Solves the task using only pre-.NET 11 APIs (JsonNamingPolicy.CamelCase, standard JsonSerializer.Serialize)"
- "Does NOT load or reference the system-text-json-net11 skill — none of its APIs are needed here"
- "Targets net8.0 and produces camelCase JSON output"
tests/dotnet11/system-text-json-net11/eval.yaml:69
- Scenario 2 also asks to run the program, but there is no
expect_toolsrequirement. Consider addingexpect_tools: ["bash"]so the evaluation enforces actually running the sample code.
- name: "Type-safe JsonTypeInfo access without exceptions in .NET 11"
prompt: |
In a .NET 11 library I'm working on, I have a `JsonSerializerOptions` instance
and I want to get the JSON metadata for a specific type `T` in a strongly-typed
way (i.e. I want back a `JsonTypeInfo<T>`, not a non-generic `JsonTypeInfo` that
I have to cast).
I also need a way to *probe* whether metadata for `T` is available without
having an exception thrown at me if it isn't — so I can branch on "have it"
vs "don't have it". I'd rather not wrap things in try/catch.
Show me a minimal `net11.0` program that demonstrates both: getting the typed
metadata when it's available, and checking-without-throwing for a case where
it might not be. Run the program.
assertions:
- type: "exit_success"
- type: "output_contains"
value: "GetTypeInfo<"
- type: "output_contains"
value: "TryGetTypeInfo<"
- type: "output_matches"
pattern: "out\\s+(var|JsonTypeInfo)"
- type: "output_matches"
pattern: "net11\\.0"
- type: "output_not_matches"
pattern: "catch\\s*\\(\\s*\\w*Exception"
rubric:
- "Uses the new generic JsonSerializerOptions.GetTypeInfo<T>() overload (added in .NET 11) for the typed-access part"
- "Uses the new generic JsonSerializerOptions.TryGetTypeInfo<T>(out JsonTypeInfo<T>? info) overload (added in .NET 11) for the probing part"
- "Does NOT wrap GetTypeInfo in try/catch as a workaround"
- "Targets net11.0 and actually runs the program"
timeout: 180
tests/dotnet11/system-text-json-net11/eval.yaml:68
- Scenario 2 rubric says the sample should demonstrate both (1) typed metadata access when available and (2) a non-throwing probe when not available. Current assertions only check for method names (
GetTypeInfo<,TryGetTypeInfo<) which can appear in printed code even if the runtime behavior isn’t demonstrated. Add assertions that verify the program output shows both branches/results (e.g., a success message plus a clear "not available"/false case).
- type: "exit_success"
- type: "output_contains"
value: "GetTypeInfo<"
- type: "output_contains"
value: "TryGetTypeInfo<"
- type: "output_matches"
pattern: "out\\s+(var|JsonTypeInfo)"
- type: "output_matches"
pattern: "net11\\.0"
- type: "output_not_matches"
pattern: "catch\\s*\\(\\s*\\w*Exception"
rubric:
- "Uses the new generic JsonSerializerOptions.GetTypeInfo<T>() overload (added in .NET 11) for the typed-access part"
- "Uses the new generic JsonSerializerOptions.TryGetTypeInfo<T>(out JsonTypeInfo<T>? info) overload (added in .NET 11) for the probing part"
- "Does NOT wrap GetTypeInfo in try/catch as a workaround"
- "Targets net11.0 and actually runs the program"
tests/dotnet11/system-text-json-net11/eval.yaml:102
- Scenario 3 prompt also asks to run the program, but the eval doesn’t require tool use. Add
expect_tools: ["bash"]so the test enforces actually executing the program.
- name: "Non-activation: camelCase JSON serialization on .NET 8"
prompt: |
I have a .NET 8 console app and I need to serialize this record to JSON with
camelCase property names:
```csharp
public record Person(string Name, int Age);
```
Show me a minimal program targeting `net8.0` that produces output like
`{"name":"Jane","age":30}` and run it.
expect_activation: false
assertions:
- type: "exit_success"
- type: "output_matches"
pattern: "net8\\.0"
- type: "output_matches"
pattern: "JsonNamingPolicy\\.CamelCase"
- type: "output_not_matches"
pattern: "JsonNamingPolicy\\.PascalCase"
- type: "output_not_matches"
pattern: "net11\\.0"
rubric:
- "Solves the task using only pre-.NET 11 APIs (JsonNamingPolicy.CamelCase, standard JsonSerializer.Serialize)"
- "Does NOT load or reference the system-text-json-net11 skill — none of its APIs are needed here"
- "Targets net8.0 and produces camelCase JSON output"
timeout: 180
- Files reviewed: 8/9 changed files
- Comments generated: 4
There was a problem hiding this comment.
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 8/9 changed files
- Comments generated: 0 new
|
@ViktorHofer and @JanKrivanek - these skills need .NET 11 installed. However, it does not seem like validator actually installs any .NET or checks if it is installed and what version it is. Is the assumption here that latest .NET will always be available on the machine? I would prefer these evaluations to not run if .NET 11 is not installed but there should be a way to install latest preview of .NET11 through some option. Thoughts? |
|
Basically conditional evals. I think someone added trait support to eval.yaml but I would need to double check. Today, in CI the evaluation.yml file installs the .NET SDK: skills/.github/workflows/evaluation.yml Lines 537 to 540 in 400661d The simplest solution might be just updating to a .NET 11 SDK |
|
@eiriktsarpalis - please review the STJ skill |
ViktorHofer
left a comment
ViktorHofer
left a comment
There was a problem hiding this comment.
Btw we already have a dotnet-upgrade plugin. Maybe that's better suited instead of the separate plugin?
Skill Coverage Report
|
There was a problem hiding this comment.
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
tests/dotnet11/system-text-json-net11/eval.yaml:101
- Scenario 3’s assertions don’t check the produced JSON payload (e.g., that it contains camelCase "name"/"age"), only that the response mentions JsonNamingPolicy.CamelCase and net8.0. This can allow false positives where the program output doesn’t match the requested JSON shape.
assertions:
- type: "exit_success"
- type: "output_matches"
pattern: "net8\\.0"
- type: "output_matches"
pattern: "JsonNamingPolicy\\.CamelCase"
- type: "output_not_matches"
pattern: "JsonNamingPolicy\\.PascalCase"
- type: "output_not_matches"
pattern: "net11\\.0"
rubric:
- "Solves the task using only pre-.NET 11 APIs (JsonNamingPolicy.CamelCase, standard JsonSerializer.Serialize)"
- "Does NOT load or reference the system-text-json-net11 skill — none of its APIs are needed here"
- "Targets net8.0 and produces camelCase JSON output"
- Files reviewed: 12/13 changed files
- Comments generated: 3
There was a problem hiding this comment.
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
eng/skill-validator/src/Evaluate/EvaluateCommand.cs:1877
IsDotnet11PluginPathuses a blanketcatch { return false; }. SincePluginDiscovery.ParsePluginJsonis documented to throw on malformed JSON so callers can surface a blocking validation error, swallowing all exceptions here can hide real problems (and can also cause dotnet11 targets to not be excluded when they should be). Prefer catching only the expected exception(s) (e.g.,JsonException) or letting unexpected exceptions propagate / be logged.
try
{
var plugin = PluginDiscovery.ParsePluginJson(pluginJsonPath);
return plugin is not null &&
string.Equals(plugin.Name, "dotnet11", StringComparison.OrdinalIgnoreCase);
}
catch { return false; }
- Files reviewed: 12/13 changed files
- Comments generated: 1
|
/evaluate |
|
/evaluate --dotnet11 |
![[Manish Jayaswal]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e302f3f-4328-44a0-845d-4eb0c930af63 Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com>
…n workflow Now that .NET 11 is the default SDK for the repo (global.json updated on main), there is no need for special dotnet11 exclusion logic. The dotnet11 plugin skills are treated like any other plugin and included in all evaluation runs. Changes: - Remove --include-dotnet11 CLI option and filtering logic from EvaluateCommand.cs - Remove IncludeDotnet11 from ValidatorConfig model - Remove --dotnet11 flag parsing from evaluation workflow gate job - Remove dotnet11 exclusion logic from discover job - Remove conditional .NET 11 preview SDK install step (global.json handles it) - Remove dotnet11 from excludeFromSchedule list - Remove all dotnet11-specific warning/status messages - Remove section 9 from InvestigatingResults.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1667ad4 to
83482b4
Compare
There was a problem hiding this comment.
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 8/9 changed files
- Comments generated: 2
|
Thanks @JanKrivanek for making .NET11 previews the default SDK. I have updated this PR to remove all changes to skill-validator and other changes related to the .NET 11 SDK. Now it is simply adding a new plugin (dotnet11) with one skill in it. @AbhitejJohn @JanKrivanek and @ViktorHofer - if you folks can approve it then I will merge it and then add the rest of dotnet11 skills to this plugin in separate PRs (so that they can be reviewed) |
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
@ViktorHofer - the requested change has been made. Can you approve this for merging? |
|
/evaluate --runs 5 |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Adding first of the several new skills for .NET11. These will cover the new/updated APIs in .NET 11