diff --git a/docs/MODEL_MANAGEMENT.md b/docs/MODEL_MANAGEMENT.md new file mode 100644 index 000000000..c5377c4ef --- /dev/null +++ b/docs/MODEL_MANAGEMENT.md @@ -0,0 +1,124 @@ +# Model Management Guide + +## Keeping Model Lists Current + +The workflows support various LLM models from different providers. This guide explains how to keep the model lists up-to-date. + +## Quick Update Process + +```bash +# Set your OpenAI API key +export OPENAI_API_KEY="sk-..." + +# Run the update script +./scripts/update_model_list.sh +``` + +This will fetch the current list of available models from OpenAI's API and display them categorized by series. + +## Manual API Query + +You can also query the API directly: + +```bash +curl https://api.openai.com/v1/models \ + -H "Authorization: Bearer $OPENAI_API_KEY" | jq '.data[].id' | sort +``` + +## Model Selection Guidelines + +### High Quality Models (for critical evaluation) +- **o1**: Latest reasoning model from OpenAI +- **gpt-4o**: Current flagship multimodal model +- **gpt-4-turbo**: Fast, capable GPT-4 variant + +### Efficient Models (for rapid iteration) +- **o3-mini**: Small reasoning model +- **o1-mini**: Smaller reasoning model +- **gpt-4o-mini**: Fast, cost-effective GPT-4o +- **gpt-3.5-turbo**: Budget-friendly option + +### GitHub Models vs OpenAI + +**GitHub Models API** (`https://models.inference.ai.azure.com`): +- Uses GITHUB_TOKEN (no additional API key needed) +- Provides access to various models including Meta-Llama +- May have different model availability than OpenAI + +**OpenAI API** (`https://api.openai.com`): +- Requires OPENAI_API_KEY +- Direct access to all OpenAI models +- Generally has latest models first + +## Updating Workflow Configurations + +After checking current models, update these files: + +### 1. Consumer Template Workflow +**File**: `templates/consumer-repo/.github/workflows/agents-verifier.yml` + +Update the model descriptions: +```yaml +model: + description: >- + GitHub Models: [list here] | OpenAI: [list here] +``` + +Update default values for compare mode (lines ~155-156): +```javascript +core.setOutput('model', 'gpt-4o'); // High quality model +core.setOutput('model2', 'o1'); // Different high quality model +``` + +### 2. Reusable Workflow (if needed) +**File**: `.github/workflows/reusable-agents-verifier.yml` + +Update input descriptions if model lists change significantly. + +### 3. Sync Changes +After updating templates: +```bash +git add templates/consumer-repo/.github/workflows/agents-verifier.yml +git commit -m "Update model lists to current OpenAI offerings" +git push +``` + +The sync workflow will automatically propagate changes to consumer repos. + +## Testing New Models + +Before setting as defaults, test new models: + +```bash +# Test with pr_verifier locally +python scripts/langchain/pr_verifier.py \ + --repo owner/repo \ + --pr 123 \ + --mode evaluate \ + --model "new-model-name" \ + --json +``` + +Or trigger workflow manually with `workflow_dispatch` and specify the model. + +## Model Naming Notes + +- **Base names** (e.g., `gpt-4o`) point to latest stable version +- **Dated versions** (e.g., `gpt-4o-2024-08-06`) are frozen snapshots +- Use base names in workflows for automatic updates +- Use dated versions when reproducibility is critical + +## Resources + +- **OpenAI Models Docs**: https://platform.openai.com/docs/models +- **OpenAI API Reference**: https://platform.openai.com/docs/api-reference/models +- **GitHub Models**: GitHub.com → Settings → GitHub Models (for available models) + +## Scheduled Updates + +Recommend checking for model updates: +- Monthly for production stability +- Weekly if using cutting-edge features +- After OpenAI announces new releases + +Run `./scripts/update_model_list.sh` to check current availability. diff --git a/docs/plans/langchain-post-code-rollout.md b/docs/plans/langchain-post-code-rollout.md index cd18031be..a0eac687e 100644 --- a/docs/plans/langchain-post-code-rollout.md +++ b/docs/plans/langchain-post-code-rollout.md @@ -1,8 +1,8 @@ # LangChain Post-Code Production Capabilities - Evaluation & Rollout Plan -> **Date:** January 6, 2026 -> **Status:** Phase 1 Complete, Phase 2 Ready -> **Last Validation:** 2026-01-06 +> **Date:** January 7, 2026 +> **Status:** Phase 1 & 2 Deployed - Ready for Live Testing +> **Last Validation:** 2026-01-07 --- @@ -50,8 +50,11 @@ 1. **Topic Splitting** - `topic_splitter.py` is called from `agents-63-issue-intake.yml` when `apply_langchain_formatting=true`. Tested and working (created 5 issues from Issues.txt). 2. **PR Verifier Script** - `pr_verifier.py` has: - - Single-provider evaluation mode - - Multi-provider comparison mode + - Single-provider evaluation mode (✅ Tested on PR #625, #270) + - Multi-provider comparison mode (✅ Tested on PR #302) + - Model selection: GitHub Models and OpenAI (PR #626, #629 merged) + - Model2 parameter for comparing different models (PR #629 merged) + - GPT-5.2 support (PR #633 pending) - Follow-up issue creation on CONCERNS/FAIL - JSON output format - Tests: 4 test files with good coverage @@ -64,20 +67,23 @@ ### ⚠️ Implemented but Not Synced/Active in Consumer Repos -1. **Verifier Labels** - Only 3 of 7 consumer repos have `verify:checkbox`, `verify:evaluate`, `verify:compare`: +1. **Verifier Labels** - All 7 consumer repos have `verify:checkbox`, `verify:evaluate`, `verify:compare`: - ✅ Manager-Database - ✅ Template - ✅ trip-planner - - ❌ Travel-Plan-Permission - - ❌ Portable-Alpha-Extension-Model - - ❌ Trend_Model_Project - - ❌ Collab-Admin + - ✅ Travel-Plan-Permission + - ✅ Portable-Alpha-Extension-Model + - ✅ Trend_Model_Project + - ⚠️ Collab-Admin (sync PR #104 pending - has failing gate check) -2. **Format Labels** - Only 3 of 7 consumer repos have `agents:format`, `agents:formatted`: +2. **Format Labels** - All 7 consumer repos have `agents:format`, `agents:formatted`, `agents:optimize`, `agents:apply-suggestions`: - ✅ Manager-Database - ✅ Template - ✅ trip-planner - - ❌ Others + - ✅ Travel-Plan-Permission + - ✅ Portable-Alpha-Extension-Model + - ✅ Trend_Model_Project + - ⚠️ Collab-Admin (sync PR #104 pending) 3. **Updated .gitignore** - Consumer repos have old partial version, missing new entries for: - `verifier-diff-summary.md` @@ -127,46 +133,52 @@ - [x] Verify `reusable-agents-verifier.yml` is correct (done) - [x] Verify `agents-verifier.yml` template calls reusable correctly (done) - [x] `.gitignore` template has all workflow status entries (not synced by design) +- [x] Add model selection and provider choice (PR #626, #629 merged) +- [x] Add scripts/update_model_list.sh for model management (PR #633) +- [x] Add docs/MODEL_MANAGEMENT.md for model update process (PR #633) - [x] Commit any fixes to main **Step 1B: Deploy to Consumer Repos** -1. ✅ All consumer repos have verifier labels (`verify:checkbox`, `verify:evaluate`, `verify:compare`) -2. ✅ Sync workflow runs automatically on template changes (last run: ~3h ago) -3. ✅ Sync PRs merged (no open sync PRs pending) -4. Test on Manager-Database (pilot): +1. ✅ All consumer repos have verifier labels (6/7 active, Collab-Admin pending) +2. ✅ Sync workflow runs automatically on template changes +3. ✅ Sync PRs merged (except Collab-Admin #104) +4. ⏳ Test on Manager-Database (pilot): - Find a recently merged agent PR - Add `verify:evaluate` label - Verify workflow runs and posts evaluation comment - - **Note:** Requires repo write access to add labels + - **Status:** Ready for testing - requires repo write access **Validation Criteria:** -- [x] Verifier labels exist in all 7 consumer repos (verified via `--check`) -- [x] `agents-verifier.yml` deployed to consumer repos -- [ ] Live test: Workflow runs without errors -- [ ] Live test: LLM evaluation produces scores and verdict -- [ ] Live test: Comment posted on PR with evaluation results -- [ ] Live test: Follow-up issue created if verdict is CONCERNS/FAIL +- [x] Verifier labels exist in all 7 consumer repos (6 active, 1 pending) +- [x] `agents-verifier.yml` deployed to consumer repos (6/7) +- [x] Model selection working (tested PRs #625, #270, #302) +- [x] Compare mode with different models working (tested PR #302) +- [ ] Live test on fresh PR: Workflow runs without errors +- [ ] Live test on fresh PR: LLM evaluation produces scores and verdict +- [ ] Live test on fresh PR: Comment posted on PR with evaluation results +- [ ] Live test on fresh PR: Follow-up issue created if verdict is CONCERNS/FAIL ### Phase 2: Issue Formatting & Cleanup (1 Step) **Step 2A: Labels & Sync** 1. ✅ Labels created via sync workflow (`agents:format`, `agents:formatted`, `agents:optimize`, `agents:apply-suggestions`) 2. ✅ `agents-issue-optimizer.yml` is in sync manifest -3. ✅ Sync PRs merged automatically -4. Test on Manager-Database: +3. ✅ Sync PRs merged automatically (6/7 repos) +4. ⏳ Test on Manager-Database: - Create test issue with unformatted content - Add `agents:format` label - Verify issue is reformatted and `agents:formatted` label added - - **Note:** Requires repo write access to add labels + - **Status:** Ready for testing - requires repo write access **Validation Criteria:** - [x] Format labels exist in consumer repos (created by sync workflow) - [x] `agents-issue-optimizer.yml` in sync manifest - [x] `issue_formatter.py` tests passing (14 tests, fixed env var isolation) -- [ ] Live test: `agents:format` triggers workflow -- [ ] Live test: Issue body updated to AGENT_ISSUE_TEMPLATE -- [ ] Live test: `agents:formatted` label added -- [ ] Live test: Original content preserved in hidden section +- [x] Topic splitting tested and working (created 5 issues from Issues.txt) +- [ ] Live test on fresh issue: `agents:format` triggers workflow +- [ ] Live test on fresh issue: Issue body updated to AGENT_ISSUE_TEMPLATE +- [ ] Live test on fresh issue: `agents:formatted` label added +- [ ] Live test on fresh issue: Original content preserved in hidden section ### Phase 3: Archive Unused Scripts (1 Step) @@ -190,8 +202,23 @@ These scripts are fully tested (145 tests passing) but not yet integrated: | Phase | Scope | Steps | Test Repo | Status | |-------|-------|-------|-----------|--------| -| 1 | PR Verification | 2 | Manager-Database | ✅ Deployed, pending live test | +| 1 | PR Verification | 2 | Manager-Database | ✅ Deployed, tested on 3 PRs | | 2 | Issue Formatting | 1 | Manager-Database | ✅ Deployed, pending live test | | 3 | Cleanup/Archive | 1 | N/A | Deferred (scripts retained) | -**Total: 4 deployment actions** - All infrastructure deployed. Live testing requires repo write access. +**Total: 4 deployment actions** - All infrastructure deployed. Collab-Admin sync PR pending. + +--- + +## Remaining Tasks + +### Immediate (Ready Now) +1. **Merge PR #633** - GPT-5.2 for compare mode (waiting for CI) +2. **Resolve Collab-Admin sync** - PR #104 blocked by failing gate check +3. **Live test `agents:format`** - Create unformatted issue, add label, verify formatting +4. **Live test `agents:optimize`** - Test on existing issue to get suggestions + +### Future Enhancements +1. **Compare mode refinement** - Currently uses gpt-4o (GitHub) vs gpt-5.2 (OpenAI) +2. **Model auto-update** - Use `scripts/update_model_list.sh` periodically +3. **Phase 3 scripts** - Decide on capability_check.py and task_decomposer.py integration diff --git a/scripts/update_model_list.sh b/scripts/update_model_list.sh new file mode 100755 index 000000000..45f83d7c8 --- /dev/null +++ b/scripts/update_model_list.sh @@ -0,0 +1,150 @@ +#!/bin/bash +# Update model list from OpenAI API and GitHub Models API +# Run this periodically to keep model information current +# Requires: OPENAI_API_KEY and/or GITHUB_TOKEN environment variables + +set -e + +echo "======================================================================" +echo "Model List Update - $(date '+%Y-%m-%d %H:%M:%S')" +echo "======================================================================" +echo "" + +# Check GitHub Models first (always available in workflows) +if [ -n "$GITHUB_TOKEN" ]; then + echo "Fetching GitHub Models..." + curl -s https://models.inference.ai.azure.com/models \ + -H "Authorization: Bearer $GITHUB_TOKEN" | \ + python3 << 'PYEOF' +import json +import sys + +try: + data = json.load(sys.stdin) + if isinstance(data, list): + models = sorted([m.get('id', m.get('name', str(m))) for m in data]) + print("GitHub Models API:") + print("-" * 60) + # Extract just the model names (after last /) + gpt_models = [] + llama_models = [] + other_models = [] + + for m in models: + if '/models/' in m: + parts = m.split('/models/') + if len(parts) > 1: + model_name = parts[1].split('/')[0] + if 'gpt' in model_name.lower(): + gpt_models.append(model_name) + elif 'llama' in model_name.lower(): + llama_models.append(model_name) + else: + other_models.append(model_name) + + if gpt_models: + print("GPT Models:") + for m in sorted(set(gpt_models)): + print(f" - {m}") + print() + + if llama_models: + print("Meta Llama Models:") + for m in sorted(set(llama_models)): + print(f" - {m}") + print() + + if other_models: + print("Other Models:") + for m in sorted(set(other_models)): + print(f" - {m}") + print() +except Exception as e: + print(f"Error fetching GitHub Models: {e}", file=sys.stderr) +PYEOF +else + echo "GitHub Models: GITHUB_TOKEN not set (skipped)" + echo "" +fi + +# Check OpenAI models +if [ -z "$OPENAI_API_KEY" ]; then + echo "OpenAI Models: OPENAI_API_KEY not set (skipped)" + echo "Get your API key from: https://platform.openai.com/account/api-keys" + echo "" +else + echo "Fetching OpenAI models..." + curl -s https://api.openai.com/v1/models \ + -H "Authorization: Bearer $OPENAI_API_KEY" | \ + python3 << 'PYEOF' +import json +import sys + +try: + data = json.load(sys.stdin) + + if 'error' in data: + print(f"Error: {data['error']['message']}", file=sys.stderr) + sys.exit(1) + + models = sorted([m['id'] for m in data['data']]) + + # Categorize models for easier review + gpt_4_models = [m for m in models if m.startswith('gpt-4') and not m.startswith('gpt-4o')] + gpt_4o_models = [m for m in models if m.startswith('gpt-4o')] + gpt_3_models = [m for m in models if m.startswith('gpt-3')] + o_models = [m for m in models if m.startswith('o1') or m.startswith('o3')] + other_models = [m for m in models if not any(m.startswith(p) for p in ['gpt-4', 'gpt-3', 'o1', 'o3'])] + + print("OpenAI API:") + print("-" * 60) + + if o_models: + print("O-Series (Reasoning Models):") + for m in o_models: + print(f" - {m}") + print() + + if gpt_4o_models: + print("GPT-4o (Omni) Series:") + for m in gpt_4o_models: + print(f" - {m}") + print() + + if gpt_4_models: + print("GPT-4 Series:") + for m in gpt_4_models: + print(f" - {m}") + print() + + if gpt_3_models: + print("GPT-3.5 Series:") + for m in gpt_3_models: + print(f" - {m}") + print() + + if other_models: + print("Other Models:") + for m in other_models: + print(f" - {m}") + print() +except Exception as e: + print(f"Error: {e}", file=sys.stderr) +PYEOF +fi + +echo "======================================================================" +echo "" +echo "Recommendations for workflow configuration:" +echo " - High Quality: o1, gpt-4o, gpt-4-turbo" +echo " - Efficient: o3-mini (if available), o1-mini, gpt-4o-mini, gpt-3.5-turbo" +echo "" +echo "Note: Models with suffixes like -YYYY-MM-DD are dated snapshots" +echo " Base names (e.g., 'gpt-4o') point to the latest stable version" +echo "" +echo "To update workflow configurations:" +echo " 1. Review the model list above" +echo " 2. Update templates/consumer-repo/.github/workflows/agents-verifier.yml" +echo " 3. Update .github/workflows/reusable-agents-verifier.yml if needed" +echo " 4. Commit and push changes" +echo " 5. Sync to consumer repos via sync workflow" diff --git a/templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml b/templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml index 93f208327..277022de9 100644 --- a/templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml +++ b/templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml @@ -62,9 +62,12 @@ jobs: echo "should_run=false" >> "$GITHUB_OUTPUT" fi - - name: Checkout repository + - name: Checkout Workflows repository if: steps.check.outputs.should_run == 'true' uses: actions/checkout@v4 + with: + repository: stranske/Workflows + ref: main - name: Set up Python if: steps.check.outputs.should_run == 'true' diff --git a/templates/consumer-repo/.github/workflows/agents-verifier.yml b/templates/consumer-repo/.github/workflows/agents-verifier.yml index a29803978..b5362f1f7 100644 --- a/templates/consumer-repo/.github/workflows/agents-verifier.yml +++ b/templates/consumer-repo/.github/workflows/agents-verifier.yml @@ -37,12 +37,21 @@ on: - compare default: 'evaluate' model: - description: 'LLM model (e.g., gpt-4o-mini, gpt-4o, o1). Run List LLM Models first' + description: >- + Model for evaluation. GitHub Models: gpt-4o, gpt-4o-mini, + text-embedding-3-large, text-embedding-3-small, Meta-Llama-3.1-405B-Instruct, + Meta-Llama-3.1-70B-Instruct, Meta-Llama-3-70B-Instruct | + OpenAI: gpt-5.2, o1, o1-preview, o1-mini, o3-mini (if available), gpt-4o, gpt-4o-mini, + gpt-4-turbo, gpt-4, gpt-3.5-turbo. Use script scripts/update_model_list.sh + to check current availability. required: false type: string default: 'gpt-4o-mini' model2: - description: 'Second model for compare mode (e.g., gpt-5, gpt-4.1)' + description: >- + Second model for compare mode. High quality options: + GitHub Models: gpt-4o, Meta-Llama-3.1-405B-Instruct | + OpenAI: gpt-5.2, o1, gpt-4o, gpt-4-turbo. Efficient: gpt-4o-mini, o1-mini required: false type: string default: '' @@ -146,7 +155,14 @@ jobs: core.info(`Verifier triggered with mode: ${mode}`); core.setOutput('should_run', 'true'); core.setOutput('mode', mode); - core.setOutput('model', ''); // Use default + // For compare mode, use high-quality models from different providers + if (mode === 'compare') { + core.setOutput('model', 'gpt-4o'); // GitHub Models - current flagship + core.setOutput('model2', 'gpt-5.2'); // OpenAI - GPT-5.2 + } else { + core.setOutput('model', ''); // Use default + core.setOutput('model2', ''); + } core.setOutput('provider', ''); // Use default core.setOutput('pr_number', pr.number.toString());