feat: mipro v2

fix: issues with exaamples

Author: dosco

.cspell/project-words.txt

@@ -7,6 +7,7 @@ APIURL
 asig
 astro
 astrojs
+bootstrapper
 Buena
 chrono
 chunker
@@ -24,6 +25,7 @@ dedup
 deepseek
 distractor
 dosco
+fewshot
 fres
 frontmatter
 fstate
@@ -43,8 +45,11 @@ logprobs
 Logprobs
 Macbook
 MCPHTTP
+minibatch
 minilm
+mipro
 Mixtral
+modelcontextprotocol
 modelinfo
 multiservice
 nanos
@@ -82,17 +87,20 @@ Sukiyabashi
 superintelligence
 textembedding
 tika
+trainset
 tsimp
 tsup
 upsert
 upserted
 usecase
 uuidv
+valset
 vectordb
 vectorize
 Vikram
 weaviate
 Weaviate
 xlarge
 xstate
-modelcontextprotocol
+minibatching
+minibatches

.vscode/settings.json

@@ -6,5 +6,6 @@
   "typescript.enablePromptUseWorkspaceTsdk": true,
   "typescript.format.enable": true,
   "typescript.tsserver.watchOptions": {},
-  "js/ts.implicitProjectConfig.target": "ESNext"
+  "js/ts.implicitProjectConfig.target": "ESNext",
+  "biome.enabled": false
 }

README.md

@@ -558,7 +558,7 @@ const res = await gen.forward({ text });
 }
 ```
 
-## Tuning the prompts (programs)
+## Tuning the prompts (Basic)
 
 You can tune your prompts using a larger model to help them run more efficiently and give you better results. This is done by using an optimizer like `AxBootstrapFewShot` with and examples from the popular `HotPotQA` dataset. The optimizer generates demonstrations `demos` which when used with the prompt help improve its efficiency.
 
@@ -628,6 +628,137 @@ const res = await program.forward({
 console.log(res);
 ```
 
+## Tuning the prompts (Advanced, Mipro v2)
+
+MiPRO v2 is an advanced prompt optimization framework that uses Bayesian optimization to automatically find the best instructions, demonstrations, and examples for your LLM programs. By systematically exploring different prompt configurations, MiPRO v2 helps maximize model performance without manual tuning.
+
+### Key Features
+
+- **Instruction optimization**: Automatically generates and tests multiple instruction candidates
+- **Few-shot example selection**: Finds optimal demonstrations from your dataset
+- **Smart Bayesian optimization**: Uses UCB (Upper Confidence Bound) strategy to efficiently explore configurations
+- **Early stopping**: Stops optimization when improvements plateau to save compute
+- **Program and data-aware**: Considers program structure and dataset characteristics
+
+### Basic Usage
+
+```typescript
+import { AxAI, AxChainOfThought, AxMiPRO } from '@ax-llm/ax'
+
+// 1. Setup your AI service
+const ai = new AxAI({
+  name: 'google-gemini',
+  apiKey: process.env.GOOGLE_APIKEY
+})
+
+// 2. Create your program
+const program = new AxChainOfThought(`input -> output`)
+
+// 3. Configure the optimizer
+const optimizer = new AxMiPRO({
+  ai,
+  program,
+  examples: trainingData, // Your training examples
+  options: {
+    numTrials: 20,  // Number of configurations to try
+    auto: 'medium'  // Optimization level
+  }
+})
+
+// 4. Define your evaluation metric
+const metricFn = ({ prediction, example }) => {
+  return prediction.output === example.output
+}
+
+// 5. Run the optimization
+const optimizedProgram = await optimizer.compile(metricFn, {
+  valset: validationData  // Optional validation set
+})
+
+// 6. Use the optimized program
+const result = await optimizedProgram.forward(ai, { input: "test input" })
+```
+
+### Configuration Options
+
+MiPRO v2 provides extensive configuration options:
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `numCandidates` | Number of instruction candidates to generate | 5 |
+| `numTrials` | Number of optimization trials | 30 |
+| `maxBootstrappedDemos` | Maximum number of bootstrapped demonstrations | 3 |
+| `maxLabeledDemos` | Maximum number of labeled examples | 4 |
+| `minibatch` | Use minibatching for faster evaluation | true |
+| `minibatchSize` | Size of evaluation minibatches | 25 |
+| `earlyStoppingTrials` | Stop if no improvement after N trials | 5 |
+| `minImprovementThreshold` | Minimum score improvement threshold | 0.01 |
+| `programAwareProposer` | Use program structure for better proposals | true |
+| `dataAwareProposer` | Consider dataset characteristics | true |
+| `verbose` | Show detailed optimization progress | false |
+
+### Optimization Levels
+
+You can quickly configure optimization intensity with the `auto` parameter:
+
+```typescript
+// Light optimization (faster, less thorough)
+const optimizedProgram = await optimizer.compile(metricFn, { auto: 'light' })
+
+// Medium optimization (balanced)
+const optimizedProgram = await optimizer.compile(metricFn, { auto: 'medium' })
+
+// Heavy optimization (slower, more thorough)
+const optimizedProgram = await optimizer.compile(metricFn, { auto: 'heavy' })
+```
+
+### Advanced Example: Sentiment Analysis
+
+```typescript
+// Create sentiment analysis program
+const classifyProgram = new AxChainOfThought<
+  { productReview: string },
+  { label: string }
+>(`productReview -> label:string "positive" or "negative"`)
+
+// Configure optimizer with advanced settings
+const optimizer = new AxMiPRO({
+  ai,
+  program: classifyProgram,
+  examples: trainingData,
+  options: {
+    numCandidates: 3,
+    numTrials: 10,
+    maxBootstrappedDemos: 2,
+    maxLabeledDemos: 3,
+    earlyStoppingTrials: 3,
+    programAwareProposer: true,
+    dataAwareProposer: true,
+    verbose: true
+  }
+})
+
+// Run optimization and save the result
+const optimizedProgram = await optimizer.compile(metricFn, {
+  valset: validationData
+})
+
+// Save configuration for future use
+const programConfig = JSON.stringify(optimizedProgram, null, 2)
+await fs.promises.writeFile('./optimized-config.json', programConfig)
+```
+
+### How It Works
+
+MiPRO v2 works through these steps:
+1. Generates various instruction candidates
+2. Bootstraps few-shot examples from your data
+3. Selects labeled examples directly from your dataset
+4. Uses Bayesian optimization to find the optimal combination
+5. Applies the best configuration to your program
+
+By exploring the space of possible prompt configurations and systematically measuring performance, MiPRO v2 delivers optimized prompts that maximize your model's effectiveness.
+
 ## Built-in Functions
 
 | Function           | Name               | Description                                  |
@@ -658,8 +789,6 @@ OPENAI_APIKEY=openai_key npm run tsx ./src/examples/marketing.ts
 | rag-docs.ts         | Convert PDF to text and embed for rag search            |
 | react.ts            | Use function calling and reasoning to answer questions  |
 | agent.ts            | Agent framework, agents can use other agents, tools etc |
-| qna-tune.ts         | Use an optimizer to improve prompt efficiency           |
-| qna-use-tuned.ts    | Use the optimized tuned prompts                         |
 | streaming1.ts       | Output fields validation while streaming                |
 | streaming2.ts       | Per output field validation while streaming             |
 | streaming3.ts       | End-to-end streaming example `streamingForward()`       |
@@ -671,7 +800,9 @@ OPENAI_APIKEY=openai_key npm run tsx ./src/examples/marketing.ts
 | simple-classify.ts  | Use a simple classifier to classify stuff               |
 | mcp-client-memory.ts       | Example of using an MCP server for memory with Ax |
 | mcp-client-blender.ts       | Example of using an MCP server for Blender with Ax |
-
+| tune-bootstrap.ts         | Use bootstrap optimizer to improve prompt efficiency           |
+| tune-mipro.ts         | Use mipro v2 optimizer to improve prompt efficiency           |
+| tune-usage.ts    | Use the optimized tuned prompts                         |
 
 ## Our Goal