Skip to content

Rebrand Azure OpenAI to Azure AI Foundry - #1922

Merged
aantn merged 8 commits into
masterfrom
claude/rename-azure-openai-foundry-TVcRs
Apr 19, 2026
Merged

aantn merged 8 commits into
masterfrom
claude/rename-azure-openai-foundry-TVcRs

Conversation

@aantn

@aantn aantn commented Apr 16, 2026 •

Copy link
Copy Markdown
Collaborator

This PR updates all references to "Azure OpenAI Service" to "Azure AI Foundry" across the codebase and documentation, reflecting Microsoft's rebranding of the service.

Summary

Azure OpenAI Service has been rebranded to Azure AI Foundry. This change updates all user-facing documentation, configuration examples, and internal references to use the new name while maintaining full backward compatibility with existing configurations.

Key Changes

  • Documentation:

    • Renamed docs/ai-providers/azure-openai.md to docs/ai-providers/azure-ai-foundry.md
    • Converted old azure-openai.md to a redirect stub to preserve external links
    • Updated all references in installation guides, multi-provider docs, and navigation files
  • Code Updates:

    • Updated comments and docstrings in holmes/core/llm.py, holmes/core/azure_token.py, and holmes/common/env_vars.py to reference "Azure AI Foundry"
    • Updated error messages and logging in tests/llm/utils/classifiers.py
  • Configuration & Setup:

    • Updated CLI installation guide to reference "Azure AI Foundry"
    • Updated Kubernetes installation examples
    • Updated contributing guidelines with new service name
  • Navigation & Metadata:

    • Updated docs/ai-providers/.nav.yml and mkdocs.yml to point to new documentation file
    • Updated docs/ai-providers/index.md to reference Azure AI Foundry

Implementation Details

  • The old azure-openai.md file is preserved as a redirect page using HTML meta refresh to avoid breaking external links
  • All environment variables (AZURE_API_KEY, AZURE_API_BASE, etc.) remain unchanged for backward compatibility
  • The actual API integration and functionality remain identical; this is purely a naming update

https://claude.ai/code/session_014a8UZvSnBVz9cbFizEL9De

Summary by CodeRabbit

  • Documentation

    • Rebranded Azure provider from "Azure OpenAI" to "Azure AI Foundry" across docs, CLI help, examples, tests, and site navigation.
    • Added a comprehensive Azure AI Foundry guide (auth options, Kubernetes/workload identity, token behavior, examples, troubleshooting) and a redirect from the old page.
    • Clarified OpenAI-compatible guidance to explicitly include LiteLLM Proxy; removed an RBAC troubleshooting section.
  • Bug Fixes

    • Removed an Azure-specific validation special-case to rely on standard environment validation.

claude added 2 commits April 15, 2026 10:55
Microsoft has rebranded Azure OpenAI Service to Azure AI Foundry.
This updates user-facing references (docs, log messages, help text,
and code comments) to use the new product name. Environment variable
names, Azure role names, API paths, and endpoint domains remain
unchanged because those are still the canonical Azure identifiers.

- Rename docs/ai-providers/azure-openai.md to azure-ai-foundry.md
- Update nav entries in mkdocs.yml and .nav.yml
- Update provider pages, installation guides, env-var reference,
  eval docs, CONTRIBUTING, and benchmark CLI help text
- Update log messages and code comments in holmes core and tests

Signed-off-by: Claude <noreply@anthropic.com>
The page was renamed to azure-ai-foundry.md in the previous commit, but
any existing external links to holmesgpt.dev/ai-providers/azure-openai/
would 404. Add a minimal stub with a meta-refresh so those links land
on the new page. The stub is not listed in .nav.yml, so it does not
appear in the sidebar navigation (awesome-nav only renders listed
entries).

Signed-off-by: Claude <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@github-actions

github-actions Bot commented Apr 16, 2026 •

Copy link
Copy Markdown
Contributor

📂 Previous Runs

📜 #4 · Run @ __c683aa4__ (#24535662907) — Apr 16, 21:56 UTC

✅ Results of HolmesGPT evals

Automatically triggered by commit c683aa4 on branch claude/rename-azure-openai-foundry-TVcRs

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 11/11 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost Total tokens Input Max input Output Max output Cached Non-cached Reasoning Compactions
✅ 09_crashpod 39.2s 7 12 $0.2744 147,302 144,859 24,273 2,443 642 120,290 24,569 — —
✅ 101_loki_historical_logs_pod_deleted 33.1s 4 7 $0.2185 81,784 79,795 22,179 1,989 884 57,320 22,475 — —
✅ 112_find_pvcs_by_uuid 24.3s 5 4 $0.1950 94,411 93,210 20,576 1,201 329 72,621 20,589 — —
✅ 12_job_crashing 40.8s 6 13 $0.2775 135,977 133,602 25,131 2,375 605 106,953 26,649 — —
✅ 176_network_policy_blocking_traffic_no_runbooks 38.9s 6 11 $0.2708 130,454 128,124 25,015 2,330 556 102,067 26,057 — —
✅ 227_count_configmaps_per_namespace[0] 23.5s 5 9 $0.2008 94,582 93,316 20,817 1,266 585 71,871 21,445 — —
✅ 243_pod_names_contain_service 29.2s 5 8 $0.2144 98,142 96,570 21,529 1,572 534 74,363 22,207 — —
✅ 24_misconfigured_pvc 33.9s 6 12 $0.2388 118,599 116,608 22,171 1,991 603 93,709 22,899 — —
✅ 43_current_datetime_from_prompt 4.9s 1 — $0.1085 16,983 16,855 16,855 128 128 0 16,855 — —
✅ 51_logs_summarize_errors 23.4s 4 5 $0.1856 77,034 75,937 20,902 1,097 334 55,023 20,914 — —
✅ 61_exact_match_counting 10.6s 2 1 $0.1255 34,557 34,211 17,347 346 278 16,854 17,357 — —
Total 27.5s avg 4.6 avg 8.2 avg $2.3100 1,029,825 1,013,087 25,131 16,738 884 771,071 242,016 — —
Benchmark Comparison Details

Baseline: latest ci-benchmark experiment on master

Status: Success - 48 test/model combinations loaded

Benchmark experiment:

No benchmark data available for comparison.

Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run.

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)
📜 #3 · Run @ __9c392bb__ (#24535519430) — Apr 16, 21:49 UTC

✅ Results of HolmesGPT evals

Automatically triggered by commit 9c392bb on branch claude/rename-azure-openai-foundry-TVcRs

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 11/11 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost Total tokens Input Max input Output Max output Cached Non-cached Reasoning Compactions
✅ 09_crashpod 39.5s 6 13 $0.2781 130,367 127,849 24,897 2,518 715 101,152 26,697 — —
✅ 101_loki_historical_logs_pod_deleted 56.6s 7 13 $0.3373 166,886 163,749 27,942 3,137 917 132,184 31,565 — —
✅ 112_find_pvcs_by_uuid 26.5s 5 4 $0.2083 98,276 97,025 22,348 1,251 357 74,664 22,361 — —
✅ 12_job_crashing 39.7s 6 16 $0.2872 138,818 136,330 25,912 2,488 688 108,686 27,644 — —
✅ 176_network_policy_blocking_traffic_no_runbooks 45.8s 6 13 $0.2714 125,464 122,922 23,816 2,542 716 97,017 25,905 — —
✅ 227_count_configmaps_per_namespace[0] 22.9s 4 9 $0.1919 77,166 75,921 20,864 1,245 587 54,419 21,502 — —
✅ 243_pod_names_contain_service 31.2s 5 8 $0.2200 99,423 97,712 21,620 1,711 632 75,183 22,529 — —
✅ 24_misconfigured_pvc 33.5s 5 12 $0.2461 104,794 102,576 22,998 2,218 882 77,991 24,585 — —
✅ 43_current_datetime_from_prompt 4.5s 1 — $0.1081 16,966 16,855 16,855 111 111 0 16,855 — —
✅ 51_logs_summarize_errors 23.3s 4 5 $0.1876 77,531 76,425 21,162 1,106 354 55,251 21,174 — —
✅ 61_exact_match_counting 11.0s 3 3 $0.1383 52,648 52,269 17,849 379 232 34,409 17,860 — —
Total 30.4s avg 4.7 avg 9.6 avg $2.4743 1,088,339 1,069,633 27,942 18,706 917 810,956 258,677 — —
Benchmark Comparison Details

Baseline: latest ci-benchmark experiment on master

Status: Success - 48 test/model combinations loaded

Benchmark experiment:

No benchmark data available for comparison.

Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run.

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)
📜 #2 · Run @ __420be27__ (#24534696689) — Apr 16, 21:36 UTC

✅ Results of HolmesGPT evals

Automatically triggered by commit 420be27 on branch claude/rename-azure-openai-foundry-TVcRs

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 11/11 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost Total tokens Input Max input Output Max output Cached Non-cached Reasoning Compactions
✅ 09_crashpod 31.5s 5 9 $0.2310 103,235 101,447 22,998 1,788 811 77,707 23,740 — —
✅ 101_loki_historical_logs_pod_deleted 58.7s 8 14 $0.3436 185,091 181,863 28,222 3,228 540 151,494 30,369 — —
✅ 112_find_pvcs_by_uuid 17.8s 3 3 $0.1603 55,152 54,178 18,923 974 574 35,244 18,934 — —
✅ 12_job_crashing 41.7s 6 15 $0.2828 133,117 130,441 25,100 2,676 635 103,937 26,504 — —
✅ 176_network_policy_blocking_traffic_no_runbooks 45.2s 6 12 $0.2651 123,421 120,944 23,669 2,477 578 95,800 25,144 — —
✅ 227_count_configmaps_per_namespace[0] 21.5s 4 9 $0.1940 77,099 75,871 20,838 1,228 580 53,794 22,077 — —
✅ 243_pod_names_contain_service 31.5s 5 8 $0.2152 98,737 97,081 21,544 1,656 420 75,227 21,854 — —
✅ 24_misconfigured_pvc 37.4s 6 14 $0.2627 123,350 120,864 23,588 2,486 919 96,274 24,590 — —
✅ 43_current_datetime_from_prompt 5.0s 1 — $0.1085 16,982 16,855 16,855 127 127 0 16,855 — —
✅ 51_logs_summarize_errors 23.3s 4 5 $0.1878 77,218 76,051 20,956 1,167 393 55,083 20,968 — —
✅ 61_exact_match_counting 12.4s 3 3 $0.1379 52,603 52,237 17,834 366 219 34,392 17,845 — —
Total 29.6s avg 4.6 avg 9.2 avg $2.3888 1,046,005 1,027,832 28,222 18,173 919 778,952 248,880 — —
Benchmark Comparison Details

Baseline: latest ci-benchmark experiment on master

Status: Success - 48 test/model combinations loaded

Benchmark experiment:

No benchmark data available for comparison.

Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run.

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)
📜 #1 · Run @ __e9535a4__ (#24534654455) — Apr 16, 21:28 UTC

✅ Results of HolmesGPT evals

Automatically triggered by commit e9535a4 on branch claude/rename-azure-openai-foundry-TVcRs

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 11/11 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost Total tokens Input Max input Output Max output Cached Non-cached Reasoning Compactions
✅ 09_crashpod 33.2s 5 10 $0.2411 104,465 102,334 23,326 2,131 867 78,434 23,900 — —
✅ 101_loki_historical_logs_pod_deleted 51.4s 7 13 $0.3018 159,380 156,450 26,140 2,930 894 130,295 26,155 — —
✅ 112_find_pvcs_by_uuid 17.0s 3 3 $0.1726 59,617 58,768 21,211 849 452 37,546 21,222 — —
✅ 12_job_crashing 40.4s 6 16 $0.2917 139,092 136,546 26,197 2,546 696 108,311 28,235 — —
✅ 176_network_policy_blocking_traffic_no_runbooks 38.8s 6 10 $0.2677 126,372 124,010 24,218 2,362 566 98,151 25,859 — —
✅ 227_count_configmaps_per_namespace[0] 23.4s 5 9 $0.2015 94,493 93,250 20,798 1,243 578 71,511 21,739 — —
✅ 243_pod_names_contain_service 28.5s 4 8 $0.2082 79,042 77,331 21,598 1,711 807 55,151 22,180 — —
✅ 24_misconfigured_pvc 36.9s 6 14 $0.2658 123,835 121,383 23,700 2,452 577 96,003 25,380 — —
✅ 43_current_datetime_from_prompt 4.6s 1 — $0.1086 16,986 16,855 16,855 131 131 0 16,855 — —
✅ 51_logs_summarize_errors 21.3s 4 5 $0.1854 76,919 75,819 20,861 1,100 336 54,946 20,873 — —
✅ 61_exact_match_counting 11.1s 3 3 $0.1381 52,628 52,254 17,842 374 227 34,401 17,853 — —
Total 27.9s avg 4.5 avg 9.1 avg $2.3828 1,032,829 1,015,000 26,197 17,829 894 764,749 250,251 — —
Benchmark Comparison Details

Baseline: latest ci-benchmark experiment on master

Status: Success - 48 test/model combinations loaded

Benchmark experiment:

No benchmark data available for comparison.

Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run.

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)

✅ Results of HolmesGPT evals

Automatically triggered by commit 263274e on branch claude/rename-azure-openai-foundry-TVcRs

View workflow logs

Results of HolmesGPT evals

  • ask_holmes: 11/11 test cases were successful, 0 regressions
Status Test case Time Turns Tools Cost Total tokens Input Max input Output Max output Cached Non-cached Reasoning Compactions
✅ 09_crashpod 31.8s 5 9 $0.2291 102,777 100,931 22,804 1,846 572 77,830 23,101 — —
✅ 101_loki_historical_logs_pod_deleted 39.3s 5 9 $0.2439 105,075 102,780 23,133 2,295 866 79,172 23,608 — —
✅ 112_find_pvcs_by_uuid 23.7s 4 5 $0.2018 80,428 79,143 22,354 1,285 465 56,447 22,696 — —
✅ 12_job_crashing 40.3s 6 16 $0.2932 139,579 137,080 26,209 2,499 686 108,314 28,766 — —
✅ 176_network_policy_blocking_traffic_no_runbooks 48.3s 7 17 $0.3091 154,383 151,301 25,951 3,082 779 123,749 27,552 — —
✅ 227_count_configmaps_per_namespace[0] 20.8s 4 9 $0.1887 77,106 75,872 20,840 1,234 585 55,020 20,852 — —
✅ 243_pod_names_contain_service 38.5s 6 11 $0.2535 125,890 123,591 23,339 2,299 622 100,238 23,353 — —
✅ 24_misconfigured_pvc 31.9s 5 11 $0.2360 101,408 99,307 22,275 2,101 571 75,759 23,548 — —
✅ 43_current_datetime_from_prompt 4.4s 1 — $0.1083 16,975 16,855 16,855 120 120 0 16,855 — —
✅ 51_logs_summarize_errors 21.4s 4 5 $0.1867 77,328 76,234 21,068 1,094 333 55,154 21,080 — —
✅ 61_exact_match_counting 8.5s 2 1 $0.1227 34,373 34,119 17,255 254 185 16,854 17,265 — —
Total 28.1s avg 4.5 avg 9.3 avg $2.3730 1,015,322 997,213 26,209 18,109 866 748,537 248,676 — —
Benchmark Comparison Details

Baseline: latest ci-benchmark experiment on master

Status: Success - 34 test/model combinations loaded

Benchmark experiment:

No benchmark data available for comparison.

Benchmark has no cost, total tokens, cached tokens data. Will appear after the next weekly benchmark run.

Comparison indicators:

  • ±0% — diff under 10% (within noise threshold)
  • ↑N%/↓N% — diff 10-25%
  • ↑N%/↓N% — diff over 25% (significant)
📖 Legend
Icon Meaning
✅ The test was successful
➖ The test was skipped
⚠️ The test failed but is known to be flaky or known to fail
🚧 The test had a setup failure (not a code regression)
🔧 The test failed due to mock data issues (not a code regression)
🚫 The test was throttled by API rate limits/overload
❌ The test failed and should be fixed before merging the PR
🔄 Re-run evals manually

⚠️ Warning: /eval comments always run using the workflow from master, not from this PR branch. If you modified the GitHub Action (e.g., added secrets or env vars), those changes won't take effect.

To test workflow changes, use the GitHub CLI or Actions UI instead:

gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/rename-azure-openai-foundry-TVcRs -f markers=regression -f filter=

Option 1: Comment on this PR with /eval:

/eval
tags: regression

Or with more options (one per line):

/eval
model: gpt-4o
tags: regression
id: 09_crashpod
iterations: 5

Run evals on a different branch (e.g., master) for comparison:

/eval
branch: master
tags: regression
Option Description
model Model(s) to test (default: same as automatic runs)
tags Pytest tags / markers (no default - runs all tests!)
id Eval ID / pytest -k filter (use /list to see valid eval names)
iterations Number of runs, max 10
branch Run evals on a different branch (for cross-branch comparison)

Quick re-run: Use /rerun to re-run the most recent /eval on this PR with the same parameters.

Option 2: Trigger via GitHub Actions UI → "Run workflow"

Option 3: Add PR labels to include extra evals (applies to both automatic runs and /eval comments):

Label Effect
evals-tag-<name> Run tests with tag <name> alongside regression
evals-id-<name> Run a specific eval by test ID
evals-model-<name> Override the model (use model list name, e.g. sonnet-4.5)

Examples: evals-tag-easy, evals-id-09_crashpod, evals-model-sonnet-4.5

🏷️ Valid tags

benchmark, chain-of-causation, compaction, confluence, context_window, coralogix, counting, database, datadog, datetime, db-connectors, easy, elasticsearch, embeds, fast, frontend, grafana, hard, images, integration, kafka, kubernetes, leaked-information, logs, loki, mcp, medium, metrics, network, newrelic, no-cicd, numerical, one-test, port-forward, prometheus, question-answer, regression, runbooks, slackbot, storage, token-limit, toolset-limitation, traces, transparency

🤖 Valid models

deepseek-chat, deepseek-r1-reasoner, deepseek-reasoner, deepseek-v3.2-chat, gemini-3-flash-preview, gemini-3-pro-preview, gemini-3.1-pro-preview, gpt-4.1, gpt-5.2-high-reasoning, gpt-5.3-codex, gpt-5.4, haiku-4.5, kimi-2.5, kimi-2.5-openrouter, opus-4.5, opus-4.6, qwen-next-80B-instruct, qwen-next-80B-thinking, sonnet-4.5, sonnet-4.6


Commands: /eval · /rerun · /list

CLI: gh workflow run eval-regression.yaml --repo HolmesGPT/holmesgpt --ref claude/rename-azure-openai-foundry-TVcRs -f markers=regression -f filter=

@coderabbitai

coderabbitai Bot commented Apr 16, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

Renames provider branding from "Azure OpenAI" to "Azure AI Foundry" across docs, CLI/help text, comments, and logs; adds a new detailed azure-ai-foundry docs page with a redirect from the old page; removes an Azure-specific special-case in DefaultLLM.check_llm validation logic.

Changes

Cohort / File(s) Summary
AI provider docs & navigation
docs/ai-providers/azure-ai-foundry.md, docs/ai-providers/azure-openai.md, docs/ai-providers/.nav.yml, docs/ai-providers/index.md, mkdocs.yml
Add new Azure AI Foundry doc (setup, Entra ID, AKS Workload Identity, troubleshooting); replace old azure-openai.md with a redirect stub; update nav and index entries to point to the new page.
Installation & usage docs
docs/installation/cli-installation.md, docs/installation/kubernetes-installation.md, docs/ai-providers/using-multiple-providers.md, docs/reference/environment-variables.md, docs/development/evaluations/running-evals.md, CONTRIBUTING.md
Rename displayed labels/headings and links from "Azure OpenAI" to "Azure AI Foundry". No environment variable names or config keys changed.
Code comments, logs, CLI help, tests
holmes/common/env_vars.py, holmes/core/azure_token.py, run_benchmarks_local.py, tests/llm/utils/classifiers.py
Update inline comments, log messages, CLI help text, and test/user-facing strings to reference "Azure AI Foundry" instead of "Azure OpenAI". No behavioral changes.
LLM validation logic
holmes/core/llm.py
Remove Azure-specific branch in DefaultLLM.check_llm that previously cleared AZURE_API_VERSION from missing keys when api_version present; now relies on standard validation flow (functional change to validation behavior).
Docs tweaks
docs/ai-providers/openai-compatible.md, docs/reference/troubleshooting.md
Call out LiteLLM Proxy explicitly in OpenAI-compatible doc; remove RBAC troubleshooting section and renumber subsequent sections.

Sequence Diagram(s)

(Skipped — changes are documentation and a small validation logic edit; no new multi-component control flow requiring a sequence diagram.)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • moshemorad
  • RoiGlinik
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: rebranding all references from 'Azure OpenAI' to 'Azure AI Foundry' across documentation, code comments, and configuration files.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Apr 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Docker images ready for 615589d2 (built in 4m 15s)

⚠️ Warning: does not support ARM (ARM images are built on release only - not on every PR)

Use these tags to pull the images for testing.

📋 Copy commands

⚠️ Temporary images are deleted after 30 days. Copy to a permanent registry before using them:

gcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:615589d2
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:615589d2 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:615589d2
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:615589d2
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:615589d2
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:615589d2 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:615589d2
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:615589d2

Patch Helm values in one line (choose the chart you use):

HolmesGPT chart:

helm upgrade --install holmesgpt ./helm/holmes \
  --set registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set image=holmes-dev:615589d2 \
  --set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set operator.image=holmes-operator-dev:615589d2

Robusta wrapper chart:

helm upgrade --install robusta robusta/robusta \
  --reuse-values \
  --set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.image=holmes-dev:615589d2 \
  --set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
  --set holmes.operator.image=holmes-operator-dev:615589d2

@netlify

netlify Bot commented Apr 16, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for holmes-docs ready!

Name Link
🔨 Latest commit c8b633d
🔍 Latest deploy log https://app.netlify.com/projects/holmes-docs/deploys/69e5657d36a1770008cd60c5
😎 Deploy Preview https://deploy-preview-1922--holmes-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@aantn

aantn commented Apr 16, 2026

Copy link
Copy Markdown
Collaborator Author

@claude review

@github-actions

github-actions Bot commented Apr 16, 2026 •

Copy link
Copy Markdown
Contributor

🔬 CLI Performance Benchmark

🟡 Startup Time (no LLM)

Measures holmes version execution time (imports + initialization)

Metric PR Master Change
Cold Start 11.60s 12.03s -3.6%
Warm Mean 5.61s 5.48s +2.4%
Warm Min 5.48s 5.46s
Warm Max 5.73s 5.50s

🟡 Full CLI with LLM

Measures holmes ask execution time (OpenRouter + Haiku 4.5)

Metric PR Master Change
Cold Start 11.65s 14.72s -20.8%
Warm Mean 6.91s 7.02s -1.6%
Warm Min 6.82s 6.90s
Warm Max 7.10s 7.23s

PR: 615589d2 | Master: fd1792a3 | Iterations: 5

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/ai-providers/azure-ai-foundry.md`:
- Line 106: Several fenced code blocks in the markdown are missing a language
label and use inconsistent fencing; locate the code blocks that contain the
snippet
"Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action"
and the other flagged blocks and normalize them to the repo style by replacing
the opening fence with a language-labeled fence (e.g., ```text) and ensuring
matching closing fences (```), so update the blocks around the occurrences of
that snippet and the other flagged snippets to use consistent triple-backtick
fences with explicit language labels.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f7bde25d-8f16-4c35-839f-63ee9fc2ec57

📥 Commits

Reviewing files that changed from the base of the PR and between 4c6a8c6 and 420be27.

📒 Files selected for processing (16)
  • CONTRIBUTING.md
  • docs/ai-providers/.nav.yml
  • docs/ai-providers/azure-ai-foundry.md
  • docs/ai-providers/azure-openai.md
  • docs/ai-providers/index.md
  • docs/ai-providers/using-multiple-providers.md
  • docs/development/evaluations/running-evals.md
  • docs/installation/cli-installation.md
  • docs/installation/kubernetes-installation.md
  • docs/reference/environment-variables.md
  • holmes/common/env_vars.py
  • holmes/core/azure_token.py
  • holmes/core/llm.py
  • mkdocs.yml
  • run_benchmarks_local.py
  • tests/llm/utils/classifiers.py

Comment thread docs/ai-providers/azure-ai-foundry.md
Comment thread docs/ai-providers/azure-openai.md Outdated
Comment thread holmes/core/llm.py Outdated
@aantn
aantn enabled auto-merge (squash) April 16, 2026 21:41
claude added 2 commits April 16, 2026 21:42
… dead code

Address two review comments on PR #1922:

1. The redirect stub azure-openai.md had an HTML comment before the
   YAML frontmatter delimiters, preventing MkDocs from parsing the
   title and hide directives. Move frontmatter to the top of the file.

2. Remove unreachable dead code in llm.py: the `if provider == "azure"`
   block inside the `else` branch could never execute because the
   `elif provider == "azure"` branch above already handles all Azure
   models.

Signed-off-by: Claude <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
holmes/core/llm.py (1)

87-87: Nit: grammar in renamed comment.

The phrase "LLM configurations used services like Azure AI Foundry" reads awkwardly — it appears a word is missing. Consider rewording while you're touching this line.

✏️ Proposed fix
-    # LLM configurations used services like Azure AI Foundry
+    # LLM configuration fields used by services like Azure AI Foundry
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@holmes/core/llm.py` at line 87, Fix the awkward comment above the LLM
configuration block by rewording it to read clearly; replace "LLM configurations
used services like Azure AI Foundry" with a concise phrase such as "LLM
configurations for services like Azure AI Foundry" or "LLM configuration for
services (e.g., Azure AI Foundry)"; update the comment located in
holmes/core/llm.py near the LLM configuration section so it reads grammatically
correct.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@holmes/core/llm.py`:
- Line 87: Fix the awkward comment above the LLM configuration block by
rewording it to read clearly; replace "LLM configurations used services like
Azure AI Foundry" with a concise phrase such as "LLM configurations for services
like Azure AI Foundry" or "LLM configuration for services (e.g., Azure AI
Foundry)"; update the comment located in holmes/core/llm.py near the LLM
configuration section so it reads grammatically correct.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 939d0299-6763-4b9e-983d-9a6f33d50eed

📥 Commits

Reviewing files that changed from the base of the PR and between 420be27 and 9c392bb.

📒 Files selected for processing (2)
  • docs/ai-providers/azure-openai.md
  • holmes/core/llm.py

Signed-off-by: Claude <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — both previously flagged issues (frontmatter ordering in redirect stub and dead code in llm.py) were addressed in commit 3d91175.

Extended reasoning...

Overview

Pure rebranding rename across 16 files: docs, nav config, code comments, log messages, and error strings. No logic, API, or environment variable changes.

Security risks

None. No authentication, authorization, or data-handling code was modified. Environment variable names remain unchanged.

Level of scrutiny

Low. The changes are mechanical text substitutions. The new azure-ai-foundry.md is a copy of the old azure-openai.md with updated terminology, and azure-openai.md is now a correctly-structured redirect stub (frontmatter first, then HTML comment, then meta-refresh). The llm.py diff removes a now-redundant AZURE_API_VERSION workaround that was replaced by a more general approach that strips already-set keys from missing_keys.

Other factors

All 11 regression evals pass. Bug hunting found no issues. Both concerns from my prior review were resolved.

fix asure model config

Signed-off-by: Arik Alon <alon.arik@gmail.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/reference/troubleshooting.md (1)

21-38: ⚠️ Potential issue | 🟡 Minor

Fix section numbering drift after RBAC section removal.

After renumbering Unclear Prompts to ## 3, the next heading should be ## 4. Model Issues (it’s currently ## 5), so the document remains sequential.

Suggested patch
-## 5. Model Issues
+## 4. Model Issues
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/reference/troubleshooting.md` around lines 21 - 38, The heading
numbering is off after renaming "Unclear Prompts" to "## 3"; update the
subsequent section header "## 5. Model Issues" to "## 4. Model Issues" so
headings remain sequential; locate the "Model Issues" heading in the document
and change its level number from 5 to 4.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@docs/reference/troubleshooting.md`:
- Around line 21-38: The heading numbering is off after renaming "Unclear
Prompts" to "## 3"; update the subsequent section header "## 5. Model Issues" to
"## 4. Model Issues" so headings remain sequential; locate the "Model Issues"
heading in the document and change its level number from 5 to 4.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 135c6a7d-6e54-44ed-bae9-b2b3dd63f6e6

📥 Commits

Reviewing files that changed from the base of the PR and between c683aa4 and c8b633d.

📒 Files selected for processing (4)
  • docs/ai-providers/azure-ai-foundry.md
  • docs/ai-providers/index.md
  • docs/ai-providers/openai-compatible.md
  • docs/reference/troubleshooting.md
✅ Files skipped from review due to trivial changes (3)
  • docs/ai-providers/openai-compatible.md
  • docs/ai-providers/index.md
  • docs/ai-providers/azure-ai-foundry.md

@aantn
aantn merged commit b24a252 into master Apr 19, 2026
20 of 22 checks passed
@aantn
aantn deleted the claude/rename-azure-openai-foundry-TVcRs branch April 19, 2026 23:33
Comment thread holmes/core/llm.py
Comment on lines 368 to 373
model_requirements = litellm.validate_environment(
model=model, api_key=api_key, api_base=api_base
)
# validate_environment does not accept api_version, and as a special case for Azure OpenAI Service,
# when all the other AZURE environments are set expect AZURE_API_VERSION, validate_environment complains
# the missing of it even after the api_version is set.
# TODO: There's an open PR in litellm to accept api_version in validate_environment, we can leverage this
# change if accepted to ignore the following check.
# https://github.com/BerriAI/litellm/pull/13808
if (
provider == "azure"
and ["AZURE_API_VERSION"] == model_requirements["missing_keys"]
and api_version is not None
):
model_requirements["missing_keys"] = []
model_requirements["keys_in_environment"] = True

if not model_requirements["keys_in_environment"]:
raise Exception(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new azure-ai-foundry.md docs show AZURE_AD_TOKEN_AUTH=true working with anthropic/claude-opus-4-7 (labeled 'recommended'), but the code does not support this workflow. Two code paths fail: (1) check_llm() only skips AZURE_API_KEY validation inside the elif provider=="azure" branch — for anthropic/ models, litellm returns provider="anthropic", so the else branch calls validate_environment without AZURE_AD_TOKEN_AUTH awareness, raising a startup error requiring ANTHROPIC_API_KEY; (2) completion() only injects the azure_ad_token when litellm_model_name.startswith("azure/"), so Anthropic prefix models never receive the Entra ID token. The documented Entra ID workflow with Anthropic models on Azure AI Foundry is completely non-functional.

Extended reasoning...

What the bug is and how it manifests

The PR adds docs/ai-providers/azure-ai-foundry.md documenting two distinct use-cases for AZURE_AD_TOKEN_AUTH=true: (a) traditional azure/ models, and (b) anthropic/claude-opus-4-7 models accessed via the Azure AI Foundry Anthropic endpoint (https://XXXX.services.ai.azure.com/anthropic). The Anthropic option is labeled "recommended." However, the code only implements AZURE_AD_TOKEN_AUTH support for the azure/ prefix model format in both the startup validation path and the request execution path.

The specific code paths that fail

Failure 1 — startup validation in check_llm() (holmes/core/llm.py): The provider detection via litellm.get_llm_provider("anthropic/claude-opus-4-7") returns provider="anthropic", not "azure". The AZURE_AD_TOKEN_AUTH-aware code that removes AZURE_API_KEY from missing_keys lives exclusively inside the elif provider == "azure" branch. When provider="anthropic", execution falls to the else branch which calls litellm.validate_environment(model=model, api_key=api_key, api_base=api_base) with no AZURE_AD_TOKEN_AUTH awareness. With api_key=None and no ANTHROPIC_API_KEY set in the environment, validate_environment reports ANTHROPIC_API_KEY as a missing key, and check_llm raises an exception immediately at startup.

Failure 2 — token injection in completion() (holmes/core/llm.py): The azure_ad_token injection guard is: if AZURE_AD_TOKEN_AUTH and litellm_model_name.startswith("azure/"). For model="anthropic/claude-opus-4-7", the name starts with "anthropic/", so the condition is False, azure_ad_kwargs stays empty, and no azure_ad_token is passed to litellm. Even if startup validation somehow passed, actual API calls would go unauthenticated to the Foundry endpoint.

Why existing code does not prevent it

The AZURE_AD_TOKEN_AUTH feature was implemented only for azure/ prefix models. The new azure-ai-foundry.md docs introduce a new use-case — Anthropic models hosted on Azure AI Foundry — that requires the same Entra ID token flow but uses a different litellm provider string. The code was never extended to cover this case.

Impact

Users who follow the documented Entra ID workflow for Anthropic models on Azure AI Foundry will see a startup failure: "model anthropic/claude-opus-4-7 requires the following environment variables: ['ANTHROPIC_API_KEY']". This is a misleading error — the user is intentionally not providing an Anthropic API key, as the docs explicitly say "No AZURE_API_KEY is needed" and the Workload Identity examples show no api_key in the modelList entries. The entire documented feature is non-functional.

Step-by-step proof

  1. User sets: AZURE_AD_TOKEN_AUTH=true, AZURE_API_BASE=https://XXXX.services.ai.azure.com/anthropic, and runs: holmes ask ... --model="anthropic/claude-opus-4-7"
  2. DefaultLLM.init calls check_llm("anthropic/claude-opus-4-7", api_key=None, ...)
  3. litellm.get_llm_provider("anthropic/claude-opus-4-7") returns provider="anthropic"
  4. Code enters else branch: model_requirements = litellm.validate_environment(model="anthropic/claude-opus-4-7", api_key=None, api_base=api_base)
  5. validate_environment finds no ANTHROPIC_API_KEY in environment, returns {"missing_keys": ["ANTHROPIC_API_KEY"], "keys_in_environment": False}
  6. check_llm raises: Exception("model anthropic/claude-opus-4-7 requires the following environment variables: ['ANTHROPIC_API_KEY']")
  7. HolmesGPT fails to start. No request is ever made.

How to fix

Extend both code paths to also handle AZURE_AD_TOKEN_AUTH when the provider is "anthropic" but an Azure AI Foundry endpoint (api_base containing "ai.azure.com" or "azure") is configured. In check_llm, add a condition that skips ANTHROPIC_API_KEY validation when AZURE_AD_TOKEN_AUTH=True and the api_base suggests an Azure endpoint. In completion(), change the guard from startswith("azure/") to also trigger for anthropic/ models when AZURE_AD_TOKEN_AUTH is enabled.

Comment on lines 144 to 150

See [OpenAI Configuration](../ai-providers/openai.md) for more details.

=== "Azure OpenAI"
=== "Azure AI Foundry"

1. **Set up API key**:
```bash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The 'Azure AI Foundry' quick-start tab in cli-installation.md still shows the legacy openai.azure.com endpoint and the outdated 2024-02-15-preview API version, even though the tab label was renamed from 'Azure OpenAI'. The new azure-ai-foundry.md guide (linked via 'See Azure AI Foundry Configuration') uses cognitiveservices.azure.com and 2025-04-01-preview, creating a direct contradiction that will confuse new Azure AI Foundry users.

Extended reasoning...

What the bug is and how it manifests

The PR renames the quick-start tab from '=== "Azure OpenAI"' to '=== "Azure AI Foundry"' in cli-installation.md (and similarly in kubernetes-installation.md), but leaves the example configuration inside unchanged. Specifically, the updated cli-installation.md still exports:

export AZURE_API_VERSION="2024-02-15-preview"
export AZURE_API_BASE="https://your-resource.openai.azure.com"

The specific code path that triggers it

A user reading the quick-start guide follows the 'Azure AI Foundry' tab, copies the example configuration, and uses the legacy openai.azure.com endpoint with the 2024-02-15-preview API version. They are then directed to 'See Azure AI Foundry Configuration' (azure-ai-foundry.md), which shows entirely different endpoint patterns: https://XXXX.services.ai.azure.com/anthropic for Anthropic models and https://YYYY.cognitiveservices.azure.com/ with api_version: 2025-04-01-preview for Azure OpenAI-style deployments.

Why existing code does not prevent it

This is an incomplete rebrand: the PR updated the tab label and the 'see more' link, but did not update the inline example configuration to match the new authoritative guide. The old endpoint domain (openai.azure.com) and old API version (2024-02-15-preview) are only valid for the legacy Azure OpenAI Service pattern, which is exactly what the PR is deprecating from a documentation perspective.

What the impact would be

New Azure AI Foundry users (who have cognitiveservices.azure.com or services.ai.azure.com resources) will copy a configuration that does not work for their resource type. Users who already have legacy Azure OpenAI resources (openai.azure.com) will not notice a functional problem, but will be confused when the detailed guide they are directed to shows completely different endpoints. This creates an inconsistent onboarding experience for the renamed service.

How to fix it

Update the quick-start example in cli-installation.md (and kubernetes-installation.md) to match the authoritative azure-ai-foundry.md guide. The simplest fix is to show the Anthropic model path as the recommended option (matching the 'recommended' callout in azure-ai-foundry.md), with a note that Azure OpenAI-style deployments use a different endpoint and API version (2025-04-01-preview, cognitiveservices.azure.com).

Step-by-step proof

  1. User opens cli-installation.md and clicks the 'Azure AI Foundry' quick-start tab.
  2. They see: export AZURE_API_VERSION="2024-02-15-preview" and export AZURE_API_BASE="https://your-resource.openai.azure.com".
  3. They follow the 'See Azure AI Foundry Configuration' link to azure-ai-foundry.md.
  4. The new guide recommends https://XXXX.services.ai.azure.com/anthropic (no API version needed) or https://YYYY.cognitiveservices.azure.com/ with api_version: 2025-04-01-preview -- neither matches the quick-start.
  5. The user is left with contradictory instructions between the quick-start tab and the detailed guide it explicitly links to.

@@ -18,26 +18,7 @@ HolmesGPT can't access logs, metrics, or traces from your observability stack.
- Verify toolset configuration connects to Prometheus/Grafana/logs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The troubleshooting doc now has sections numbered 1, 2, 3, 5 — skipping 4. When removing the old '## 3. RBAC Permissions' section and renumbering '## 4. Unclear Prompts' to '## 3. Unclear Prompts', the subsequent '## 5. Model Issues' was not renumbered to '## 4. Model Issues'. Fix by updating the heading to '## 4. Model Issues'.

Extended reasoning...

What the bug is and how it manifests

The PR removes the old '## 3. RBAC Permissions' section from docs/reference/troubleshooting.md and correctly renumbers old '## 4. Unclear Prompts' to '## 3. Unclear Prompts'. However, the final section '## 5. Model Issues' was left unchanged, resulting in a document with sections numbered 1, 2, 3, 5 — a visible gap at position 4.

The specific code path that triggers it

The renumbering was performed on the section heading at the deletion site (changing 4→3) but not propagated to the next downstream heading. The diff shows:

  • removed
  • → (correctly renumbered)
  • — unchanged (should have become )

Why existing code doesn't prevent it

There is no automated enforcement of sequential section numbers in Markdown documents. The author manually renumbered one heading but missed the next one.

What the impact would be

Readers navigating the troubleshooting guide will see the sequence 1→2→3→5 and either assume content is missing between 3 and 5 or lose confidence in the document's accuracy. It's a minor cosmetic issue but degrades documentation quality.

How to fix it

Change to in docs/reference/troubleshooting.md.

Step-by-step proof

  1. Before PR: sections were 1 (Truncation), 2 (Missing Data Access), 3 (RBAC Permissions), 4 (Unclear Prompts), 5 (Model Issues).
  2. PR removes section 3 and renumbers 4→3: now 1, 2, 3 (Unclear Prompts), 5 (Model Issues).
  3. Section 5 was not decremented to 4, leaving the gap.
  4. The file as merged contains the heading at what is now logically the 4th section.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants