feat: add Base14 Scout toolset integration - #1860
Closed
nimisha-gj wants to merge 14 commits into
Closed
nimisha-gj wants to merge 14 commits into
nimisha-gj wants to merge 14 commits into
Conversation
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
…override Signed-off-by: nimishgj <nimishgj444@gmail.com>
…ubclasses - Replace username/password auth with client_id/client_secret (client_credentials grant) - Fix toolset_manager to eagerly init RemoteMCPToolset subclasses via isinstance check so Scout tools are rediscovered from cache instead of being empty - Update tests to match new auth fields Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
Signed-off-by: nimishgj <nimishgj444@gmail.com>
…nology Signed-off-by: nimishgj <nimishgj444@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
WalkthroughThis PR introduces a new Base14 Scout observability platform integration to HolmesGPT as a built-in Remote MCP toolset. Changes include documentation updates describing the integration, core toolset manager modifications to handle RemoteMCPToolset instances, the Scout toolset implementation with OAuth client-credentials support, LLM instruction template, and test coverage. Changes
Sequence DiagramsequenceDiagram
participant Scout as ScoutToolset
participant OAuth as OAuth/OIDC Server
participant MCP as MCP Server
participant API as Base14 Scout API
Scout->>Scout: Load config from env vars & YAML
alt Using OAuth credentials
Scout->>OAuth: Fetch .well-known/openid-configuration
OAuth-->>Scout: OAuth metadata (token_endpoint)
Scout->>OAuth: POST client_credentials grant
OAuth-->>Scout: access_token
Scout->>Scout: Store Bearer token
else Using API token
Scout->>Scout: Use api_token directly
end
Scout->>MCP: Connect with Bearer Authorization header
MCP-->>Scout: Connection established
Scout->>MCP: Discover available tools
MCP-->>Scout: Tool definitions (health, services, spans, logs, etc.)
Scout->>API: Ready to execute MCP tools
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
docs/data-sources/builtin-toolsets/base14-scout.md (1)
99-118: Consider removing the detailed tools list.Per coding guidelines, toolset documentation should not list what a toolset/integration can do because "users discover capabilities by using Holmes, and feature lists become stale quickly." The tools are auto-discovered from the MCP server, so maintaining this list creates a documentation maintenance burden.
Consider simplifying to:
## Available Tools -All tools are auto-discovered from the Scout MCP server. The following tools are typically available: - -| Tool | Description | -|------|-------------| -| `get_system_health_summary` | Full system overview — alerts, services, and performance stats | -| `list_services` | Discover all known services | -| `get_service_topology` | Service-to-service dependency map | -| `get_service_dependencies` | Upstream/downstream dependencies for a specific service | -| `get_service_profile` | Error rates and latency stats per relationship | -| `get_service_metrics` | Metrics emitted by a service | -| `get_last_n_alerts` | Recent Grafana alert state transitions | -| `discover_spans` | Available span types and attributes for a service | -| `discover_logs` | Available log attributes and severity levels | -| `discover_metrics` | Available metrics, types, and dimensions | -| `query_traces` | Query traces with filtering by span name, status, attributes | -| `query_trace_by_id` | Full trace detail with all spans, events, and links | -| `query_logs` | Query logs with filtering by severity, body, attributes | -| `query_metrics` | Query metric values with aggregation and grouping | +All tools are auto-discovered from the Scout MCP server at startup. Run `holmes ask "What tools are available?"` to see the current list.As per coding guidelines: "Don't list what a toolset/integration can do in documentation - users discover capabilities by using Holmes, and feature lists become stale quickly"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/data-sources/builtin-toolsets/base14-scout.md` around lines 99 - 118, Remove the detailed tools table under the "## Available Tools" section (which lists entries like `get_system_health_summary`, `list_services`, `query_traces`, etc.) and replace it with a single concise paragraph stating that tools are auto-discovered from the Scout MCP server and that users should discover available capabilities via Holmes; include the guideline note: "Don't list what a toolset/integration can do in documentation - users discover capabilities by using Holmes, and feature lists become stale quickly." Ensure the header "## Available Tools" remains but the long table is removed to avoid stale documentation.tests/test_scout_toolset.py (2)
27-29: Use specific exception type instead of genericException.The test should catch the specific exception raised by Pydantic validation (
ValidationError) rather than a blindException. This ensures the test validates the correct failure mode.♻️ Proposed fix
+from pydantic import ValidationError + class TestScoutConfig: ... def test_api_url_required(self): - with pytest.raises(Exception): + with pytest.raises(ValidationError): ScoutConfig(api_token="test-token")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/test_scout_toolset.py` around lines 27 - 29, Change the test to assert the specific Pydantic validation error: in test_api_url_required replace the generic with pytest.raises(ValidationError) so the test catches the Pydantic failure from constructing ScoutConfig without api_url; import ValidationError from pydantic and ensure the test references test_api_url_required and ScoutConfig to locate the change.
70-71: Prefix unused variable with underscore.The
msgvariable is unpacked but never used. Prefix it with an underscore to indicate it's intentionally ignored.♻️ Proposed fix
- ok, msg = toolset.prerequisites_callable({}) + ok, _msg = toolset.prerequisites_callable({}) assert ok is False🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/test_scout_toolset.py` around lines 70 - 71, The test unpacks the return of toolset.prerequisites_callable({}) into ok, msg but never uses msg; update the unpacking to prefix the unused variable with an underscore (e.g., ok, _msg = toolset.prerequisites_callable({})) in tests/test_scout_toolset.py so the intent is clear and linters won’t flag the unused variable; keep the call to toolset.prerequisites_callable unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@holmes/plugins/toolsets/scout/toolset_scout.py`:
- Around line 70-75: The code currently builds a Keycloak-specific token URL
(`token_url = f"{auth_server_url}/protocol/openid-connect/token"`) which breaks
OIDC interoperability; change the logic in the function that reads `metadata`
(using `well_known_url`, `auth_servers`, `auth_server_url`) to derive
`token_url` from the OIDC discovery document instead: prefer
metadata.get("token_endpoint") (or metadata["token_endpoint"] when present) and
only fall back to a computed URL if absolutely necessary; update uses of
`token_url` accordingly and remove the hardcoded Keycloak path so the function
truly "derives the token endpoint" per OpenID Connect discovery.
---
Nitpick comments:
In `@docs/data-sources/builtin-toolsets/base14-scout.md`:
- Around line 99-118: Remove the detailed tools table under the "## Available
Tools" section (which lists entries like `get_system_health_summary`,
`list_services`, `query_traces`, etc.) and replace it with a single concise
paragraph stating that tools are auto-discovered from the Scout MCP server and
that users should discover available capabilities via Holmes; include the
guideline note: "Don't list what a toolset/integration can do in documentation -
users discover capabilities by using Holmes, and feature lists become stale
quickly." Ensure the header "## Available Tools" remains but the long table is
removed to avoid stale documentation.
In `@tests/test_scout_toolset.py`:
- Around line 27-29: Change the test to assert the specific Pydantic validation
error: in test_api_url_required replace the generic with
pytest.raises(ValidationError) so the test catches the Pydantic failure from
constructing ScoutConfig without api_url; import ValidationError from pydantic
and ensure the test references test_api_url_required and ScoutConfig to locate
the change.
- Around line 70-71: The test unpacks the return of
toolset.prerequisites_callable({}) into ok, msg but never uses msg; update the
unpacking to prefix the unused variable with an underscore (e.g., ok, _msg =
toolset.prerequisites_callable({})) in tests/test_scout_toolset.py so the intent
is clear and linters won’t flag the unused variable; keep the call to
toolset.prerequisites_callable unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f6e20031-f0fe-45b8-96f1-9edc583feb14
⛔ Files ignored due to path filters (1)
images/integration_logos/base14-scout-icon.pngis excluded by!**/*.png
📒 Files selected for processing (11)
README.mddocs/data-sources/builtin-toolsets/.nav.ymldocs/data-sources/builtin-toolsets/base14-scout.mddocs/data-sources/builtin-toolsets/index.mddocs/why-holmesgpt.mdholmes/core/toolset_manager.pyholmes/plugins/toolsets/__init__.pyholmes/plugins/toolsets/scout/__init__.pyholmes/plugins/toolsets/scout/toolset_scout.jinja2holmes/plugins/toolsets/scout/toolset_scout.pytests/test_scout_toolset.py
Comment on lines
+70
to
+75
| auth_servers = metadata.get("authorization_servers", []) | ||
| if not auth_servers: | ||
| raise ValueError(f"No authorization_servers found in well-known metadata at {well_known_url}") | ||
|
|
||
| auth_server_url = auth_servers[0].rstrip("/") | ||
| token_url = f"{auth_server_url}/protocol/openid-connect/token" |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What is the standard OAuth 2.0 OIDC discovery endpoint for finding the token_endpoint?
💡 Result:
The standard OAuth 2.0 OIDC discovery endpoint for finding the token_endpoint is the OpenID Provider Configuration endpoint, located at /.well-known/openid-configuration appended to the Issuer URL (e.g., https://example.com/.well-known/openid-configuration). This JSON document includes the token_endpoint field, as defined in the OpenID Connect Discovery 1.0 specification.
Citations:
- 1: https://openid.net/specs/openid-connect-discovery-1_0.html.
- 2: https://swagger.io/docs/specification/v3_0/authentication/openid-connect-discovery/
- 3: https://docs.authlib.org/en/latest/oauth/oidc/discovery.html
- 4: https://openid.net/specs/openid-connect-discovery-1_0.html
- 5: https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc
🏁 Script executed:
cat -n holmes/plugins/toolsets/scout/toolset_scout.py | head -100Repository: HolmesGPT/holmesgpt
Length of output: 4069
🏁 Script executed:
# Get the full context of the _obtain_token_via_client_credentials function
sed -n '50,90p' holmes/plugins/toolsets/scout/toolset_scout.pyRepository: HolmesGPT/holmesgpt
Length of output: 1647
🏁 Script executed:
# Check what imports are used for HTTP requests
head -40 holmes/plugins/toolsets/scout/toolset_scout.pyRepository: HolmesGPT/holmesgpt
Length of output: 1195
Hardcoded token endpoint breaks compatibility with non-Keycloak OAuth providers.
Line 75 constructs the token URL using the Keycloak-specific path /protocol/openid-connect/token, which won't work with Auth0, Okta, Azure AD, or other OAuth2 providers. The function's docstring claims to "derive the token endpoint" but instead hardcodes it.
Per the OpenID Connect Discovery specification, the token endpoint should be discovered from the OIDC configuration document at {auth_server}/.well-known/openid-configuration, which includes the standard token_endpoint field.
Proposed fix using standard OIDC discovery
auth_servers = metadata.get("authorization_servers", [])
if not auth_servers:
raise ValueError(f"No authorization_servers found in well-known metadata at {well_known_url}")
auth_server_url = auth_servers[0].rstrip("/")
- token_url = f"{auth_server_url}/protocol/openid-connect/token"
+
+ # Discover token endpoint from OIDC configuration
+ oidc_config_url = f"{auth_server_url}/.well-known/openid-configuration"
+ logger.debug(f"Fetching OIDC configuration from {oidc_config_url}")
+ oidc_resp = http_requests.get(oidc_config_url, verify=verify_ssl, timeout=10)
+ oidc_resp.raise_for_status()
+ oidc_config = oidc_resp.json()
+
+ token_url = oidc_config.get("token_endpoint")
+ if not token_url:
+ raise ValueError(f"No token_endpoint found in OIDC configuration at {oidc_config_url}")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@holmes/plugins/toolsets/scout/toolset_scout.py` around lines 70 - 75, The
code currently builds a Keycloak-specific token URL (`token_url =
f"{auth_server_url}/protocol/openid-connect/token"`) which breaks OIDC
interoperability; change the logic in the function that reads `metadata` (using
`well_known_url`, `auth_servers`, `auth_server_url`) to derive `token_url` from
the OIDC discovery document instead: prefer metadata.get("token_endpoint") (or
metadata["token_endpoint"] when present) and only fall back to a computed URL if
absolutely necessary; update uses of `token_url` accordingly and remove the
hardcoded Keycloak path so the function truly "derives the token endpoint" per
OpenID Connect discovery.
Signed-off-by: nimishgj <nimishgj444@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
base14/scouttoolset for the Base14 Scout observability platformRemoteMCPToolset— tools are auto-discovered from the Scout MCP server at startupChanges
New files:
holmes/plugins/toolsets/scout/— ScoutConfig, ScoutToolset, LLM investigation instructionstests/test_scout_toolset.py— 9 unit testsdocs/data-sources/builtin-toolsets/base14-scout.md— full documentation pageimages/integration_logos/base14-scout-icon.png— logoModified files:
holmes/plugins/toolsets/__init__.py— register ScoutToolset in plugin loaderholmes/core/toolset_manager.py— fix eager init forRemoteMCPToolsetsubclasses (isinstance check)README.md— add Scout to data sources tabledocs/data-sources/builtin-toolsets/index.md— add grid carddocs/data-sources/builtin-toolsets/.nav.yml— add nav entrydocs/why-holmesgpt.md— add Scout to integration categoriesAuthentication
Supports two auth modes:
Toolset Manager Fix
RemoteMCPToolsetsubclasses (likeScoutToolset) are registered asToolsetType.BUILTINbyload_builtin_toolsets(), which means they were lazily initialized from cache — tools weren't rediscovered. Addedisinstance(toolset, RemoteMCPToolset)check alongside the existingtoolset.type == ToolsetType.MCPcheck so any MCP-based toolset gets eagerly initialized.Checklist
config_classesdefined with Field annotations for JSON Schema generationTest plan
poetry run pytest tests/test_scout_toolset.py -v --no-covSummary by CodeRabbit
New Features
Documentation