Add policy-based filtering for tool calls

docs/development/evaluations/history/results_20260120_143709.md

@@ -117,4 +117,4 @@ Status of all evaluations across models. Color coding:
 | [96_no_matching_runbook](https://github.com/HolmesGPT/holmesgpt/blob/master/tests/llm/fixtures/test_ask_holmes/96_no_matching_runbook/test_case.yaml) [🔗](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22metadata.eval_id%2520%253D%2520%252296_no_matching_runbook%2522%22%2C%20%22label%22%3A%20%22metadata.eval_id%2520equals%252096_no_matching_runbook%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) | [🟢 100% (5/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-haiku-4-5-20251001-v1%253A0%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-haiku-4-5-20251001-v1%253A0%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 54.8s / 💰 $0.10 | [🟡 80% (4/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-opus-4-5-20251101-v1%253A0%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-opus-4-5-20251101-v1%253A0%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 56.1s / 💰 $0.33 | [🟢 100% (5/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-sonnet-4-5-20250929-v1%253A0%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-sonnet-4-5-20250929-v1%253A0%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 60.8s / 💰 $0.30 | [🟡 60% (3/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bopenai%252Fgpt-5.2%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bopenai%252Fgpt-5.2%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 59.5s / 💰 $0.19 |
 
 ---
-*Results are automatically generated and updated weekly. View full traces and detailed analysis in [Braintrust experiment: ci-benchmark-21173789819](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819).*
+*Results are automatically generated and updated weekly. View full traces and detailed analysis in [Braintrust experiment: ci-benchmark-21173789819](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21173789819).*

docs/development/evaluations/history/results_20260122_074259.md

@@ -123,4 +123,4 @@ Status of all evaluations across models. Color coding:
 | [96_no_matching_runbook](https://github.com/HolmesGPT/holmesgpt/blob/master/tests/llm/fixtures/test_ask_holmes/96_no_matching_runbook/test_case.yaml) [🔗](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22metadata.eval_id%2520%253D%2520%252296_no_matching_runbook%2522%22%2C%20%22label%22%3A%20%22metadata.eval_id%2520equals%252096_no_matching_runbook%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) | [🟢 100% (5/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-haiku-4-5-20251001-v1%253A0%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-haiku-4-5-20251001-v1%253A0%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 47.8s / 💰 $0.10 | [🟡 40% (2/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-opus-4-5-20251101-v1%253A0%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-opus-4-5-20251101-v1%253A0%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 61.0s / 💰 $0.35 | [🟢 100% (5/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-sonnet-4-5-20250929-v1%253A0%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bbedrock%252Feu.anthropic.claude-sonnet-4-5-20250929-v1%253A0%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 54.1s / 💰 $0.26 | [🟡 20% (1/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bgemini%252Fgemini-3-flash-preview%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bgemini%252Fgemini-3-flash-preview%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 198.1s / 💰 $0.10 | [🟡 40% (2/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bgemini%252Fgemini-3-pro-preview%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bgemini%252Fgemini-3-pro-preview%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 112.8s / 💰 $0.28 | [🟡 40% (2/5)](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400?c=&search=%7B%22filter%22%3A%20%5B%7B%22text%22%3A%20%22span_attributes.name%2520%253D%2520%252296_no_matching_runbook%255Bopenai%252Fgpt-5.2%255D%2522%22%2C%20%22label%22%3A%20%22Name%2520equals%252096_no_matching_runbook%255Bopenai%252Fgpt-5.2%255D%22%2C%20%22originType%22%3A%20%22form%22%7D%5D%7D) / ⏱️ 41.9s / 💰 $0.12 |
 
 ---
-*Results are automatically generated and updated weekly. View full traces and detailed analysis in [Braintrust experiment: ci-benchmark-21238150400](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400).*
+*Results are automatically generated and updated weekly. View full traces and detailed analysis in [Braintrust experiment: ci-benchmark-21238150400](https://www.braintrust.dev/app/robustadev/p/HolmesGPT/experiments/ci-benchmark-21238150400).*