Skip to content

csharp-scripts description too broad — activates on language-agnostic prompts, +375% token cost over Python baseline #255

Description

@T-Gro

The csharp-scripts skill itself is fine — dotnet file.cs is a valid and useful tool for .NET-specific tasks. The problem is the skill's description is over-broad: words like "script", "experimentation", "prototyping", "quick", "without creating a full project" match virtually any throwaway coding task, causing the skill to activate even when the prompt has no .NET dependency. Once activated, the agent switches from Python (the natural choice for throwaway computations) to dotnet file.cs, which involves SDK resolution, compilation, and NuGet restore overhead — burning significantly more tokens for equal or worse results.

Eval results (claude-opus-4.6, 3 runs)

Baseline (no skill) With csharp-scripts
Language chosen Python C#
Tokens 39,973 189,020 (+375%)
Quality (rubric) 5.0/5 4.9/5
Quality (overall) 5.0/5 4.3/5
Score -15.3% (significant, CI [-37.0%, -8.0%])

Suggestion

Narrow the description to trigger only when the user explicitly wants C# — e.g. replace "quick experimentation, prototyping, and concept testing" with "quick C# experimentation" or "testing a C# language feature". The skill should activate when someone asks to run C# code, not when they just want "a quick script".

Context

Discovered while evaluating a dotnet-fsi-interactive skill for interactive .NET REPL usage. During A/B testing, csharp-scripts consistently over-activated on prompts that had no C#/.NET requirement, outcompeting both the baseline (Python) and the REPL skill with worse token efficiency.

eval.yaml used for reproduction
scenarios:
  - name: "Quick script to prototype a calendar computation"
    prompt: >
      Write a quick script to experiment with finding the earliest Friday the
      13th after year 0 in the Gregorian calendar. Also count how many Friday
      the 13ths exist in a full 400-year cycle. Execute it and show results —
      no need for a full project, just a small script for quick prototyping.
    assertions:
      - type: "exit_success"
      - type: "output_contains"
        value: "Friday"
      - type: "output_matches"
        pattern: "(April|4/13|0001-04-13)"
      - type: "output_contains"
        value: "688"
    expect_tools: ["bash"]
    max_turns: 10
    rubric:
      - "The agent correctly finds April 13, 0001 as the earliest Friday the 13th"
      - "The agent correctly counts 688 Friday the 13ths in 400 years"
      - "The agent verified by running code"
    timeout: 120

Run with: dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- --runs 3 --tests-dir tests/dotnet plugins/dotnet/skills/csharp-scripts

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions