diff --git a/plugins/dotnet-ai/skills/mlnet/SKILL.md b/plugins/dotnet-ai/skills/mlnet/SKILL.md new file mode 100644 index 0000000000..ccf961c7cf --- /dev/null +++ b/plugins/dotnet-ai/skills/mlnet/SKILL.md @@ -0,0 +1,202 @@ +--- +name: mlnet +description: > + USE FOR: Classical ML on structured data (classification, regression, clustering, anomaly + detection, recommendation, time-series forecasting). Deep learning tasks (image classification, + object detection, NER, QA, text classification, sentence similarity) via TorchSharp/TensorFlow + integration. Also for extending ML.NET pipelines with custom transforms. + DO NOT USE FOR: Natural language generation/understanding with LLMs (use meai-chat-integration), + running pre-trained ONNX models standalone without ML.NET (use onnx-runtime-inference). + For custom neural network architectures beyond what ML.NET trainers provide, use + TorchSharp directly (see references/torchsharp.md). +--- + +# ML.NET Model Training & Deployment + +Train and deploy ML.NET models for classical ML and deep learning tasks in .NET. + +## What ML.NET Covers + +| Category | Tasks | Powered By | +|---|---|---| +| **Classical ML** | Classification, regression, clustering, anomaly detection, recommendation, time-series forecasting | Built-in trainers (SDCA, LightGBM, FastTree, etc.) | +| **Deep Learning** | Image classification, object detection, NER, QA, text classification, sentence similarity | TorchSharp (`Microsoft.ML.TorchSharp`) | +| **Pre-trained Models** | TensorFlow model scoring, ONNX model scoring | TensorFlow.NET (⚠️ pinned to TF 2.3.1), ONNX Runtime | +| **AutoML** | Automated model/hyperparameter selection across all supported tasks | `Microsoft.ML.AutoML` | + +## Inputs + +| Input | Description | +|-------|-------------| +| Task description | What the model should predict or detect | +| Data description | CSV file, database source, or in-memory collection | +| Target column | The column the model predicts (label) | +| Existing project context | Current solution, target framework, existing dependencies | + +## Workflow + +### Step 1 — Install Packages + +Install the core package and any task-specific packages: + +``` +dotnet add package Microsoft.ML +``` + +- **IF recommendation:** also `dotnet add package Microsoft.ML.Recommender` +- **IF time-series:** also `dotnet add package Microsoft.ML.TimeSeries` +- **IF image classification, object detection, NER, QA, text classification, or sentence similarity:** also `dotnet add package Microsoft.ML.TorchSharp` and the appropriate `libtorch` runtime package (e.g., `libtorch-cpu` or `libtorch-cuda-12.1`) +- **IF consuming a TensorFlow model:** also `dotnet add package Microsoft.ML.TensorFlow` — ⚠️ TensorFlow support in ML.NET is pinned to TF 2.3.1 (via TensorFlow.NET 0.20.1). This works for scoring legacy TF models but may not support models from newer TF versions. For new deep learning work, prefer the TorchSharp-backed trainers above. +- **IF consuming an ONNX model within ML.NET:** also `dotnet add package Microsoft.ML.OnnxTransformer` +- **IF text featurization for classical ML:** included in the core `Microsoft.ML` package + +### Step 2 — Create MLContext + +ALWAYS create with a seed for reproducibility: + +```csharp +var mlContext = new MLContext(seed: 0); +``` + +### Step 3 — Load Data + +**IF CSV:** + +```csharp +IDataView data = mlContext.Data.LoadFromTextFile( + path: "data.csv", + hasHeader: true, + separatorChar: ','); +``` + +**IF DataFrame needed for complex data preparation:** read `references/dataframe.md` before proceeding. + +**IF in-memory collection:** + +```csharp +IDataView data = mlContext.Data.LoadFromEnumerable(records); +``` + +### Step 4 — Select Task & Build Pipeline + +Choose the trainer based on the task: + +**Classical ML (structured data)** + +| Task | Trainer | +|------|---------| +| Binary yes/no classification | `mlContext.BinaryClassification.Trainers.SdcaLogisticRegression()` | +| Multi-category classification | `mlContext.MulticlassClassification.Trainers.SdcaMaximumEntropy()` | +| Numeric prediction (regression) | `mlContext.Regression.Trainers.Sdca()` | +| Grouping (clustering) | `mlContext.Clustering.Trainers.KMeans(numberOfClusters: N)` | +| Anomaly detection | `mlContext.AnomalyDetection.Trainers.RandomizedPca()` | +| Recommendation | `mlContext.Recommendation().Trainers.MatrixFactorization()` | +| Time-series forecasting | `mlContext.Forecasting.ForecastBySsa()` | + +**Deep Learning (TorchSharp-backed)** + +| Task | Trainer | Package | +|------|---------|---------| +| Image classification | `mlContext.MulticlassClassification.Trainers.ImageClassification()` | `Microsoft.ML.TorchSharp` | +| Object detection | `mlContext.MulticlassClassification.Trainers.ObjectDetection()` | `Microsoft.ML.TorchSharp` | +| Text classification | `mlContext.MulticlassClassification.Trainers.TextClassification()` | `Microsoft.ML.TorchSharp` | +| Named entity recognition | `mlContext.MulticlassClassification.Trainers.NamedEntityRecognition()` | `Microsoft.ML.TorchSharp` | +| Question answering | `mlContext.MulticlassClassification.Trainers.QuestionAnswer()` | `Microsoft.ML.TorchSharp` | +| Sentence similarity | `mlContext.Regression.Trainers.SentenceSimilarity()` | `Microsoft.ML.TorchSharp` | + +Prepend transforms to the pipeline before the trainer: + +```csharp +var pipeline = mlContext.Transforms.Text.FeaturizeText("TextFeatures", "TextColumn") + .Append(mlContext.Transforms.Categorical.OneHotEncoding("CategoryEncoded", "CategoryColumn")) + .Append(mlContext.Transforms.Concatenate("Features", "TextFeatures", "CategoryEncoded", "NumericColumn")) + .Append(mlContext.Transforms.NormalizeMinMax("Features")) + .Append(trainer); +``` + +Use text featurization (`FeaturizeText`), one-hot encoding, concatenation into a single `Features` column, and normalization as needed. + +### Step 5 — Train + +```csharp +var model = pipeline.Fit(trainingData); +``` + +### Step 6 — Evaluate + +Split data into training and test sets: + +```csharp +var split = mlContext.Data.TrainTestSplit(data, testFraction: 0.2); +var model = pipeline.Fit(split.TrainSet); +var predictions = model.Transform(split.TestSet); +``` + +Use task-appropriate metrics: + +| Task | Metric call | Key metric | +|------|-------------|------------| +| Binary classification | `mlContext.BinaryClassification.Evaluate(predictions)` | Accuracy, AUC | +| Multiclass classification | `mlContext.MulticlassClassification.Evaluate(predictions)` | MicroAccuracy, LogLoss | +| Regression | `mlContext.Regression.Evaluate(predictions)` | RSquared, MAE | +| Clustering | `mlContext.Clustering.Evaluate(predictions)` | AverageDistance | +| Anomaly detection | `mlContext.AnomalyDetection.Evaluate(predictions)` | AUC | + +### Step 7 — Save & Use + +Save the trained model: + +```csharp +mlContext.Model.Save(model, trainingData.Schema, "model.zip"); +``` + +Create a prediction engine for inference: + +```csharp +var engine = mlContext.Model.CreatePredictionEngine(model); +var result = engine.Predict(new ModelInput { /* ... */ }); +``` + +> ⚠️ `PredictionEngine` is NOT thread-safe. In ASP.NET Core, use `PredictionEnginePool`: + +```csharp +// In Startup/Program.cs: +builder.Services.AddPredictionEnginePool() + .FromFile("model.zip"); + +// In a controller/service: +public class PredictionController(PredictionEnginePool pool) +{ + public ModelOutput Predict(ModelInput input) => pool.Predict(input); +} +``` + +Pre-warm the pool at startup to avoid cold-start latency on the first request. + +### Step 8 — Custom Transforms (Conditional) + +IF the user needs a custom pipeline step (domain-specific featurizer, external embedding lookup, custom encoder), read `references/custom-transforms.md` before proceeding. + +### Step 9 — Custom Neural Networks with TorchSharp (Conditional) + +IF the user needs a custom neural network architecture beyond what ML.NET's built-in trainers provide, read `references/torchsharp.md` before proceeding. TorchSharp provides full PyTorch bindings for .NET and can be used standalone or alongside ML.NET. + +## Validation + +- [ ] Model trains without errors +- [ ] Evaluation metrics are reasonable for the task +- [ ] Prediction engine produces output on sample input +- [ ] Model saved to disk (`model.zip` exists) + +## Pitfalls + +- **Not setting MLContext seed** — results become non-reproducible across runs. +- **Using an LLM for classification on structured data** — slower and more expensive than ML.NET for tasks like classification, regression, and clustering on structured data. +- **Not evaluating on a held-out test set** — overfitting goes undetected. +- **Forgetting to concatenate features** — all feature columns must be combined into a single `Features` column before training. +- **Not normalizing numeric features** — distance-based algorithms (KMeans, PCA) perform poorly on unnormalized data. +- **Using Accord.NET for new projects** — Accord.NET is archived and unmaintained. Use ML.NET instead. + +## More Info + +https://learn.microsoft.com/dotnet/machine-learning/ diff --git a/plugins/dotnet-ai/skills/mlnet/references/custom-transforms.md b/plugins/dotnet-ai/skills/mlnet/references/custom-transforms.md new file mode 100644 index 0000000000..67f55da52e --- /dev/null +++ b/plugins/dotnet-ai/skills/mlnet/references/custom-transforms.md @@ -0,0 +1,162 @@ +# Building Custom ML.NET Transforms + +Four approaches to extending ML.NET pipelines with custom `IEstimator` / `ITransformer` implementations. Use this reference when the agent needs to add domain-specific transforms such as embeddings, encoders, or feature engineering steps. + +## Key Problem + +ML.NET's internal base classes (`RowToRowTransformerBase`, `OneToOneTransformerBase`) have `private protected` constructors — they are **inaccessible from external projects**. You cannot subclass them outside the ML.NET repository. + +## Decision Tree + +``` +Need a custom transform? +├─ Prototyping / simple logic? ──────────────────────► Approach A (CustomMapping Lambda) +├─ Production code, external project? ───────────────► Approach B (Facade + CustomMapping) +├─ CustomMapping POCO binding too limited? ──────────► Approach C (Direct IEstimator/ITransformer) +└─ Contributing to ML.NET itself / using a fork? ───► Approach D (Source Contribution) +``` + +--- + +## Approach A — CustomMapping Lambda (Quick Start) + +Minimal code. Good for prototyping. Preserves lazy evaluation. + +```csharp +var pipeline = mlContext.Transforms.CustomMapping( + (input, output) => + { + output.TransformedFeature = input.RawFeature * 2.0f; + }, + contractName: "MyTransform"); +``` + +**Advantages:** Single line to add to a pipeline, preserves ML.NET's lazy row-by-row evaluation. + +**Limitations:** No clean public API surface, no resource lifecycle management, limited save/load support without a `CustomMappingFactory`. + +--- + +## Approach B — Production Facade + CustomMapping (Recommended) + +Wrap `CustomMapping` inside a proper `IEstimator` facade. Best balance of API quality and framework integration for external projects. + +```csharp +public class MyTransformEstimator : IEstimator +{ + private readonly MLContext _mlContext; + private readonly Lazy _resource; + + public MyTransformEstimator(MLContext mlContext, string modelPath) + { + _mlContext = mlContext ?? throw new ArgumentNullException(nameof(mlContext)); + // Lazy ensures resource is loaded once, thread-safe by default + _resource = new Lazy(() => ExpensiveResource.Load(modelPath)); + } + + public ITransformer Fit(IDataView input) + { + var resource = _resource.Value; + var pipeline = _mlContext.Transforms.CustomMapping( + (src, dst) => + { + dst.Embedding = resource.Encode(src.Text); + }, + contractName: "MyTransform"); + return pipeline.Fit(input); + } + + public SchemaShape GetOutputSchema(SchemaShape inputSchema) + { + // Validate input schema and describe output + return inputSchema; + } +} +``` + +**Save/Load pattern** — register a `CustomMappingFactory`: + +```csharp +[CustomMappingFactoryAttribute("MyTransform")] +public class MyTransformFactory : CustomMappingFactory +{ + public override Action GetMapping() + { + return (input, output) => + { + output.Embedding = DefaultResource.Encode(input.Text); + }; + } +} +``` + +**Thread safety:** `Lazy` is thread-safe by default (`LazyThreadSafetyMode.ExecutionAndPublication`). Safe for concurrent prediction engines. + +--- + +## Approach C — Direct IEstimator/ITransformer + +Implement `IEstimator` and `ITransformer` from scratch. Use when `CustomMapping` POCO binding is insufficient — for example, dynamic schemas, multi-column mapping, or variable-length output. + +```csharp +public class MyTransformer : ITransformer +{ + public bool IsRowToRowMapper => true; + + public DataViewSchema GetOutputSchema(DataViewSchema inputSchema) + { + var builder = new DataViewSchema.Builder(); + builder.AddColumns(inputSchema); + builder.AddColumn("NewFeature", NumberDataViewType.Single); + return builder.ToSchema(); + } + + public IRowToRowMapper GetRowToRowMapper(DataViewSchema inputSchema) + { + throw new NotImplementedException("Implement for row-by-row mapping"); + } + + public IDataView Transform(IDataView input) + { + // Materialize and transform data + // WARNING: This loses lazy evaluation — all rows are read into memory + return new MyDataView(input); + } + + public void Save(ModelSaveContext ctx) + { + // Serialize model parameters + } +} +``` + +**Advantages:** Maximum flexibility, no POCO binding constraints. + +**Disadvantages:** Loses lazy row-by-row evaluation (materializes data), significantly more code, must handle schema propagation manually. + +--- + +## Approach D — Source Contribution (Internal) + +Subclass `RowToRowTransformerBase` or `OneToOneTransformerBase` directly. **Only works inside the ML.NET repository or a fork.** + +```csharp +// Only compiles inside dotnet/machinelearning repo +internal sealed class MyInternalTransform : RowToRowTransformerBase +{ + public MyInternalTransform(IHostEnvironment env) + : base(env, nameof(MyInternalTransform)) + { + } + + // Full framework integration: ONNX export, zero-copy data access + protected override IRowMapper MakeRowMapper(DataViewSchema schema) + { + return new Mapper(this, schema); + } +} +``` + +**Advantages:** Full framework integration, ONNX export support, zero-copy data access via cursors. + +**Use for:** Contributing transforms upstream to the `dotnet/machinelearning` repository. diff --git a/plugins/dotnet-ai/skills/mlnet/references/dataframe.md b/plugins/dotnet-ai/skills/mlnet/references/dataframe.md new file mode 100644 index 0000000000..b86d4b09da --- /dev/null +++ b/plugins/dotnet-ai/skills/mlnet/references/dataframe.md @@ -0,0 +1,79 @@ +# DataFrame for Data Preparation in ML.NET + +Use `Microsoft.Data.Analysis.DataFrame` for loading and preparing tabular data before ML.NET training. Read this reference when the user has complex data preparation needs (filtering, grouping, column transforms, missing value handling). + +## Package + +``` +dotnet add package Microsoft.Data.Analysis +``` + +## Loading Data + +```csharp +// From CSV +DataFrame df = DataFrame.LoadCsv("data.csv"); + +// Programmatic construction +var nameColumn = new StringDataFrameColumn("Name", new[] { "Alice", "Bob" }); +var ageColumn = new Int32DataFrameColumn("Age", new[] { 30, 25 }); +var df = new DataFrame(nameColumn, ageColumn); +``` + +## Inspection + +```csharp +df.Head(5); // First 5 rows +df.Description(); // Summary statistics per column +df.Info(); // Column names, types, non-null counts +df.Rows.Count; // Row count +df.Columns.Count; // Column count +``` + +## Filtering + +```csharp +DataFrame filtered = df.Filter(df["Age"].ElementwiseGreaterThan(18)); +DataFrame subset = df.Filter(df["Category"].ElementwiseEquals("A")); +``` + +## Column Transforms + +```csharp +// Arithmetic +df["Total"] = df["Price"].Multiply(df["Quantity"]); +df["Normalized"] = df["Value"].Subtract(min).Divide(max - min); + +// Add / remove columns +df.Columns.Add(new Int32DataFrameColumn("NewCol", df.Rows.Count)); +df.Columns.Remove("UnneededColumn"); +``` + +## Missing Values + +```csharp +// Fill with default +df["Age"].FillNulls(0, inPlace: true); + +// Drop rows with any null +DataFrame clean = df.DropNulls(); +``` + +## Grouping & Aggregation + +```csharp +DataFrame grouped = df.GroupBy("Category").Sum("Amount"); +DataFrame counts = df.GroupBy("Region").Count(); +``` + +## ML.NET Integration + +`DataFrame` implements `IDataView` directly — pass it straight to ML.NET pipelines without conversion: + +```csharp +var mlContext = new MLContext(seed: 0); + +// Use DataFrame as IDataView +var split = mlContext.Data.TrainTestSplit(df, testFraction: 0.2); +var model = pipeline.Fit(split.TrainSet); +``` diff --git a/plugins/dotnet-ai/skills/mlnet/references/torchsharp.md b/plugins/dotnet-ai/skills/mlnet/references/torchsharp.md new file mode 100644 index 0000000000..16182732c6 --- /dev/null +++ b/plugins/dotnet-ai/skills/mlnet/references/torchsharp.md @@ -0,0 +1,135 @@ +# TorchSharp — Custom Neural Networks in .NET + +Read this reference when the user needs to go beyond ML.NET's built-in trainers — custom neural network architectures, training loops, or PyTorch-style development in C#/F#. + +## What TorchSharp Is + +TorchSharp is a standalone .NET library providing bindings to PyTorch's LibTorch C++ backend. It exposes PyTorch's tensor operations, neural network modules, optimizers, and data loading to .NET developers. It is **not part of ML.NET** — though ML.NET's deep learning trainers (image classification, object detection, NER, QA, text classification, sentence similarity) are powered by TorchSharp internally. + +## When to Use TorchSharp Directly (vs ML.NET) + +| Scenario | Use | +|---|---| +| Standard tasks (image classification, object detection, NER, QA, text classification) | ML.NET's TorchSharp-backed trainers (high-level pipeline API) | +| Custom neural network architectures | TorchSharp directly | +| Research / experimentation with model design | TorchSharp directly | +| Loading TorchScript models exported from Python | TorchSharp directly (`torch.jit`) | +| Computer vision utilities and pre-trained vision models | `TorchSharp.PyBridge` + `TorchVision` | +| Audio processing and models | `TorchAudio` | + +## Install + +``` +dotnet add package TorchSharp +``` + +Add a LibTorch runtime backend (pick one): + +| Scenario | Package | +|---|---| +| CPU only | `dotnet add package libtorch-cpu` | +| CUDA 11.x GPU | `dotnet add package libtorch-cuda-11.8` | +| CUDA 12.x GPU | `dotnet add package libtorch-cuda-12.1` | + +> ⚠️ The `libtorch` packages are large (~2GB for CUDA). CPU-only is sufficient for inference and small-scale training. + +## Define a Custom Model + +Inherit from `torch.nn.Module` to define custom architectures: + +```csharp +using TorchSharp; +using static TorchSharp.torch; +using static TorchSharp.torch.nn; + +public class SimpleClassifier : Module +{ + private readonly Module layers; + + public SimpleClassifier(int inputSize, int hiddenSize, int numClasses) + : base("SimpleClassifier") + { + layers = Sequential( + Linear(inputSize, hiddenSize), + ReLU(), + Dropout(0.5), + Linear(hiddenSize, numClasses)); + + RegisterComponents(); + } + + public override Tensor forward(Tensor input) + { + return layers.forward(input); + } +} +``` + +> ⚠️ ALWAYS call `RegisterComponents()` in the constructor. Without it, parameters won't be tracked by the optimizer. + +## Training Loop + +TorchSharp uses an explicit training loop (PyTorch-style), not ML.NET's `pipeline.Fit()` pattern: + +```csharp +var model = new SimpleClassifier(inputSize: 784, hiddenSize: 128, numClasses: 10); +var optimizer = torch.optim.Adam(model.parameters(), lr: 0.001); +var loss_fn = torch.nn.CrossEntropyLoss(); + +for (int epoch = 0; epoch < numEpochs; epoch++) +{ + model.train(); + foreach (var (data, target) in trainLoader) + { + optimizer.zero_grad(); + var output = model.forward(data); + var loss = loss_fn.forward(output, target); + loss.backward(); + optimizer.step(); + } +} +``` + +## Load TorchScript Models (From Python) + +Load models exported from Python via `torch.jit.save()` without needing a Python runtime: + +```csharp +var model = torch.jit.load("model.pt"); +model.eval(); + +using var input = torch.randn(1, 3, 224, 224); +using var output = model.forward(input); +``` + +## Save and Load .NET Models + +```csharp +// Save +model.save("model_weights.dat"); + +// Load +var loaded = new SimpleClassifier(784, 128, 10); +loaded.load("model_weights.dat"); +``` + +## Relationship to ML.NET + +- ML.NET's `Microsoft.ML.TorchSharp` package uses TorchSharp internally to provide high-level trainers (image classification, object detection, NER, QA, text classification, sentence similarity). +- If the built-in ML.NET trainers cover your task, prefer them — they integrate with the ML.NET pipeline API, AutoML, and Model Builder. +- Use TorchSharp directly when you need full control over the model architecture, training loop, or loss function. + +## Key Namespaces + +| Namespace | Purpose | +|---|---| +| `TorchSharp.torch` | Core tensor operations, device management | +| `TorchSharp.torch.nn` | Neural network modules (Linear, Conv2d, ReLU, etc.) | +| `TorchSharp.torch.optim` | Optimizers (Adam, SGD, etc.) | +| `TorchSharp.torch.jit` | TorchScript model loading | +| `TorchSharp.torch.utils.data` | Dataset and DataLoader abstractions | + +## More Information + +- +- diff --git a/tests/dotnet-ai/mlnet/eval.yaml b/tests/dotnet-ai/mlnet/eval.yaml new file mode 100644 index 0000000000..43cd8bcf1c --- /dev/null +++ b/tests/dotnet-ai/mlnet/eval.yaml @@ -0,0 +1,46 @@ +scenarios: + - name: "Binary classification on tabular data" + prompt: "Build an ML.NET model to predict whether a customer will churn based on this CSV data. The target column is 'Churn' (true/false)." + setup: + files: + - path: "ChurnPredictor/ChurnPredictor.csproj" + content: | + + + Exe + net10.0 + + + - path: "ChurnPredictor/Program.cs" + content: | + Console.WriteLine("TODO: Build churn prediction model"); + - path: "ChurnPredictor/data.csv" + content: | + CustomerId,Age,Income,MonthsActive,Churn + 1,25,50000,12,false + 2,45,80000,36,true + 3,35,60000,24,false + assertions: + - type: "output_contains" + value: "MLContext" + - type: "output_contains" + value: "BinaryClassification" + - type: "exit_success" + rubric: + - "Creates MLContext with a seed for reproducibility" + - "Loads data from CSV using LoadFromTextFile or DataFrame" + - "Uses BinaryClassification trainer (not multiclass or regression)" + - "Evaluates the model with a train/test split" + - "Does not suggest using an LLM for this tabular task" + timeout: 360 + + - name: "Reject LLM for tabular classification" + prompt: "I have a spreadsheet with product data and customer ratings. I want to use GPT-4 to classify products into categories based on the ratings. Can you help?" + assertions: + - type: "output_contains" + value: "ML.NET" + rubric: + - "Redirects from LLM to ML.NET for tabular classification" + - "Explains why ML.NET is better for this task (faster, cheaper, purpose-built)" + - "Provides ML.NET multiclass classification approach" + timeout: 360