Add benchmarks for Claude 4.5

.github/workflows/eval-benchmarks.yaml

@@ -115,11 +115,19 @@ jobs:
       - name: Generate benchmark report
         if: always()
         run: |
+          # Generate latest results
           poetry run python scripts/generate_eval_report.py \
             --json-file eval_results.json \
             --output-file docs/development/evaluations/latest-results.md \
             --models "${{ steps.test-command.outputs.models }}"
 
+          # Also generate timestamped version for history
+          TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+          poetry run python scripts/generate_eval_report.py \
+            --json-file eval_results.json \
+            --output-file "docs/development/evaluations/history/results_${TIMESTAMP}.md" \
+            --models "${{ steps.test-command.outputs.models }}"
+
       - name: Upload eval results
         if: always()
         uses: actions/upload-artifact@v4

docs/.nav.yml

@@ -0,0 +1,9 @@
+nav:
+  - index.md
+  - Installation: installation
+  - Walkthrough: walkthrough
+  - AI Providers: ai-providers
+  - Data Sources: data-sources
+  - Development: development
+  - Reference: reference
+  - Community: community.md

docs/ai-providers/.nav.yml

@@ -0,0 +1,12 @@
+nav:
+  - index.md
+  - Anthropic: anthropic.md
+  - AWS Bedrock: aws-bedrock.md
+  - Azure OpenAI: azure-openai.md
+  - Gemini: gemini.md
+  - Google Vertex AI: google-vertex-ai.md
+  - Ollama: ollama.md
+  - OpenAI: openai.md
+  - OpenAI-Compatible: openai-compatible.md
+  - Robusta AI: robusta-ai.md
+  - Using Multiple Providers: using-multiple-providers.md

docs/data-sources/.nav.yml

@@ -0,0 +1,6 @@
+nav:
+  - index.md
+  - Built-in Toolsets: builtin-toolsets
+  - Custom Toolsets: custom-toolsets.md
+  - Remote MCP Servers: remote-mcp-servers.md
+  - Adding Permissions for Additional Resources: permissions.md

docs/data-sources/builtin-toolsets/.nav.yml

@@ -0,0 +1,29 @@
+nav:
+  - index.md
+  - AKS Node Health: aks-node-health.md
+  - ArgoCD: argocd.md
+  - AWS: aws.md
+  - Azure Kubernetes Service: aks.md
+  - Azure SQL Database: azure-sql.md
+  - Confluence: confluence.md
+  - Coralogix logs: coralogix-logs.md
+  - DataDog: datadog.md
+  - Datetime: datetime.md
+  - Docker: docker.md
+  - GitHub: github.md
+  - Loki: grafanaloki.md
+  - Tempo: grafanatempo.md
+  - Helm: helm.md
+  - Internet: internet.md
+  - Kafka: kafka.md
+  - Kubernetes: kubernetes.md
+  - MongoDB Atlas: mongodb-atlas.md
+  - New Relic: newrelic.md
+  - Notion: notion.md
+  - OpenSearch logs: opensearch-logs.md
+  - OpenSearch status: opensearch-status.md
+  - Prometheus: prometheus.md
+  - RabbitMQ: rabbitmq.md
+  - Robusta: robusta.md
+  - ServiceNow: servicenow.md
+  - Slab: slab.md

docs/development/.nav.yml

@@ -0,0 +1,3 @@
+nav:
+  - index.md
+  - Evaluations: evaluations

docs/development/evaluations/.nav.yml

@@ -0,0 +1,7 @@
+nav:
+  - index.md
+  - Latest Results: latest-results.md
+  - Historical Results: history
+  - Running Evaluations: running-evals.md
+  - Adding New Evaluations: adding-evals.md
+  - Reporting with Braintrust: reporting.md

docs/development/evaluations/history/.nav.yml

@@ -0,0 +1,3 @@
+nav:
+  - index.md
+  - "*"

docs/development/evaluations/history/index.md

@@ -1,27 +1,5 @@
 # Historical Evaluation Results
 
-This directory contains archived evaluation results from previous runs. These results help track HolmesGPT's performance over time and identify regressions or improvements.
-
 ## Available Results
 
-- [Results from 2025-09-08 21:47:56](./results_20250908_214756.md)
-- [Results from 2025-09-08 18:19:29](./results_20250908_181929.md)
-- [Results from 2025-09-08 15:35:35](./results_20250908_153535.md)
-- [Results from 2025-09-03 22:11:55](./results_20250903_221155.md)
-
-## Understanding the Results
-
-Each result file contains:
-- Model performance metrics (pass rate, success rate)
-- Individual test outcomes
-- Execution times and resource usage
-- Comparison across different models when applicable
-
-## Automated Archiving
-
-New results are automatically added here when:
-- Weekly benchmarks run (every Sunday at 2 AM UTC)
-- Manual benchmark runs are completed
-- Significant model updates are tested
-
-For the most recent results, see the [latest results](../latest-results.md) page.
+All benchmark results are listed in the navigation sidebar.

docs/development/evaluations/history/results_20250928_001434.md

@@ -1,9 +1,9 @@
-# HolmesGPT LLM Evaluation Benchmark Results
+# September 28, 2025 - 00:14:34
 
-**Generated**: 2025-09-28 00:14 UTC
+**Generated**: 2025-09-29 10:49 UTC
 **Total Duration**: 1h 4m 41s
 **Iterations**: 1
-**Judge (classifier) model**: gpt-4o
+**Judge (classifier) model**: gpt-4.1
 
 ## About this Benchmark
 

docs/development/evaluations/index.md

@@ -66,7 +66,7 @@ Our CI/CD pipeline runs evaluations automatically:
 - **Pull Requests** - When eval-related files are modified (quick validation)
 - **On-demand** - Via GitHub Actions UI
 
-Results are published here and archived in [history/](./history/).
+Results are published here and archived in [history](./history/index.md).
 
 ## Model Comparison
 
@@ -86,4 +86,4 @@ See the [latest results](./latest-results.md) for current model performance comp
 - **[Running Evaluations](./running-evals.md)** - Complete guide to running tests
 - **[Adding New Evaluations](./adding-evals.md)** - Contribute test scenarios
 - **[Reporting with Braintrust](./reporting.md)** - Analyze results in detail
-- **[Historical Results](./history/)** - Past benchmark data
+- **[Historical Results](./history/index.md)** - Past benchmark data