dotnet-ai evaluation OOM-killed on standard runners

[danmoseley]

dotnet-ai evaluation OOM-killed on standard runners

Problem

The dotnet-ai plugin evaluation is consistently being killed by the OOM killer (exit code 137) on ubuntu-latest runners. This was first observed in scheduled run 23070747448.

The single skill in this plugin — technology-selection — has scenarios that build and run ML.NET projects (TicketClassifier, ChurnPredictor). With the scheduled evaluation parallelism settings (--parallel-scenarios 3 --parallel-runs 3), up to 9 concurrent agent runs can each dotnet build + dotnet run these ML.NET projects simultaneously, each loading training data and building models in memory. This exceeds the runner's ~7 GB RAM limit.

At the time of the OOM kill, the orphan process cleanup shows:

• 5x TicketClassifier processes
• 3x ChurnPredictor processes
• ~20 bash shells (agent sandboxes)

The scenarios responsible are:

• "ML.NET classification on tabular data" — builds and runs TicketClassifier
• "Reject LLM for tabular classification" — builds and runs ChurnPredictor

Impact

• dotnet-ai evaluation never completes successfully
• Prior to Make publish-benchmark resilient to individual plugin eval failures #365, this also blocked publish-benchmark for all plugins, preventing any dashboard updates since March 11
• No evaluation data for dotnet-ai appears on the dashboard

Suggested fix

The simplest fix would be to reduce the parallelism for these ML.NET scenarios. Some options (in order of increasing effort):

1. Reduce scenario count in eval.yaml: If some of the 6 scenarios are lower priority, reducing to 3-4 would cut peak memory.

2. Make scenarios lighter: Use smaller training datasets (tickets.csv, customers.csv) so the ML.NET models consume less memory during training/inference.

3. Avoid dotnet run in scenario setup: If the scenarios only need the agent to write the code (not execute it), removing the runtime step would eliminate the ML.NET memory overhead entirely. The agent can still dotnet build to verify compilation.

4. Request lower parallelism for this plugin: The workflow could be extended to support per-plugin parallelism overrides. For example, dotnet-ai could declare "maxParallelScenarios": 1 in its plugin.json.

cc @jeffschwMSFT @luisquintanilla — you authored the technology-selection skill in #177. Would any of the above approaches work for you?

Reproduction

The failure occurs reliably on every scheduled evaluation run since dotnet-ai was added to the repo. See:

• Run 23070747448 — evaluate (dotnet-ai) killed with exit 137

Environment

• Runner: ubuntu-latest (standard GitHub-hosted, ~7 GB RAM)
• Job timeout: 90 minutes
• Parallelism: --parallel-skills 2 --parallel-scenarios 3 --parallel-runs 3
• Exit code: 137 (SIGKILL / OOM)

[danmoseley]

The dashboard blocking impact is mitigated by #365, which makes publish-benchmark resilient to individual plugin failures.

[danmoseley]

I noticed this because https://dotnet.github.io/skills/ has stopped updating several days ago and I guess we didn't notice the regular runs were failing.
@adityamandaleeka