Add comprehensive Python SDK reference documentation - #1644
Merged
Merged
Conversation
- Fix incorrect defaults: max_steps is 40 (not 10), model has no default (not gpt-4.1) - Fix model examples to use Anthropic Claude (anthropic/claude-sonnet-4-5-20250929) - Remove non-existent tool.error field from examples - Slim installation page to install stub + quick start with link to reference - Add full API reference with correct signatures for Config, ToolCallingLLM, LLMResult, ToolCallResult - Add custom YAML toolset loading example with sample toolset file - Add Python-based custom toolset example with incident API pattern - Update mkdocs.yml and .nav.yml navigation https://claude.ai/code/session_01RuoZoiEtUGjL4MnxDec572 Signed-off-by: Claude <noreply@anthropic.com>
Contributor
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... 📒 Files selected for processing (1)
✏️ Tip: You can disable in-progress messages and the fortune message in your review settings. WalkthroughAdds a centralized Python SDK reference and replaces in-file installation examples with a pointer to it; surfaces the new doc in navigation; adds Config.support for programmatic additional toolsets and threads them into ToolsetManager with prerequisite checks. Changes
Sequence Diagram(s)sequenceDiagram
participant Config
participant ToolsetManager
participant ToolsetRegistry
participant PrereqChecker
Config->>ToolsetManager: instantiate(additional_toolsets)
ToolsetManager->>ToolsetRegistry: _list_all_toolsets()
ToolsetRegistry-->>ToolsetManager: combined toolsets (includes additional_toolsets as CUSTOMIZED)
ToolsetManager->>PrereqChecker: check_toolset_prerequisites(enabled additional toolsets)
PrereqChecker-->>ToolsetManager: prerequisites OK / errors
ToolsetManager->>ToolsetRegistry: load_toolset_with_status(selected toolset)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/reference/python-sdk.md (1)
13-17: Optional: avoid overridingmax_stepsin the minimal quick start.Setting
max_steps=15in the first example can read like a recommended default, while the API reference later states default is40. Consider omitting it (or adding a short reason) to reduce ambiguity.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/reference/python-sdk.md` around lines 13 - 17, The quick-start example sets max_steps=15 which conflicts with the documented default of 40 and may imply a recommended override; update the example in the Config instantiation by removing the max_steps argument (or replace it with a brief inline comment explaining why a non-default value would be used) so the minimal snippet only shows api_key and model and doesn’t imply a changed default; locate the Config(...) call where max_steps=15 is present and remove or comment that parameter.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@docs/reference/python-sdk.md`:
- Around line 13-17: The quick-start example sets max_steps=15 which conflicts
with the documented default of 40 and may imply a recommended override; update
the example in the Config instantiation by removing the max_steps argument (or
replace it with a brief inline comment explaining why a non-default value would
be used) so the minimal snippet only shows api_key and model and doesn’t imply a
changed default; locate the Config(...) call where max_steps=15 is present and
remove or comment that parameter.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
docs/installation/python-installation.mddocs/reference/.nav.ymldocs/reference/python-sdk.mdmkdocs.yml
- Shorten install page link to "For the full API reference" - Remove model_validator deprecated fields example - Rewrite Python toolset section: add overview of what to implement and why - Fix toolset example to match actual codebase patterns: - Tools subclass Tool with _invoke() and get_parameterized_one_liner() - Parameters use Dict[str, ToolParameter] not list of dicts - Tools return StructuredToolResult not plain strings - prerequisites_callable takes (config: dict) -> Tuple[bool, str] - Simplify example: service status + search records HTTP tools https://claude.ai/code/session_01RuoZoiEtUGjL4MnxDec572 Signed-off-by: Claude <noreply@anthropic.com>
SDK changes: - Add `additional_toolsets` parameter to Config and ToolsetManager - Additional toolsets are merged after built-in/custom YAML toolsets - Prerequisites are always checked (not cached) for additional toolsets Docs changes: - Rewrite Python toolset example with tested httpbin.org pattern - Fix _toolset assignment order (must be after super().__init__()) - Show end-to-end usage: define toolset + pass via additional_toolsets - Switch API reference from bullet lists to tables for readability - Add additional_toolsets to Config parameter table All examples tested end-to-end with OpenRouter + Claude Opus 4.6. All 1422 non-LLM tests pass. https://claude.ai/code/session_01RuoZoiEtUGjL4MnxDec572 Signed-off-by: Claude <noreply@anthropic.com>
Contributor
📂 Previous Runs📜 Run @ b70e19c (#22569201443)✅ Results of HolmesGPT evalsAutomatically triggered by commit b70e19c on branch Results of HolmesGPT evals
📜 Run @ 8d5f15d (#22566275477)✅ Results of HolmesGPT evalsAutomatically triggered by commit 8d5f15d on branch Results of HolmesGPT evals
📜 Run @ 915d1fc (#22400874312)✅ Results of HolmesGPT evalsAutomatically triggered by commit 915d1fc on branch Results of HolmesGPT evals
✅ Results of HolmesGPT evalsAutomatically triggered by commit 978a659 on branch Results of HolmesGPT evals
📖 Legend
🔄 Re-run evals manually
Option 1: Comment on this PR with Or with more options (one per line): Run evals on a different branch (e.g., master) for comparison:
Quick re-run: Use Option 2: Trigger via GitHub Actions UI → "Run workflow" Option 3: Add PR labels to include extra evals in automatic regression runs:
Examples: 🏷️ Valid tags
Commands: CLI: |
Contributor
|
✅ Docker images ready for
Use these tags to pull the images for testing. 📋 Copy commandsgcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:fe9e7e56
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:fe9e7e56 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:fe9e7e56
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:fe9e7e56
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:fe9e7e56
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:fe9e7e56 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:fe9e7e56
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:fe9e7e56Patch Helm values in one line (choose the chart you use): HolmesGPT chart: helm upgrade --install holmesgpt ./helm/holmes \
--set registry=me-west1-docker.pkg.dev/robusta-development/development \
--set image=holmes-dev:fe9e7e56 \
--set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set operator.image=holmes-operator-dev:fe9e7e56Robusta wrapper chart: helm upgrade --install robusta robusta/robusta \
--reuse-values \
--set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.image=holmes-dev:fe9e7e56 \
--set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.operator.image=holmes-operator-dev:fe9e7e56 |
Contributor
🔬 CLI Performance Benchmark🟡 Startup Time (no LLM)Measures
🟡 Full CLI with LLMMeasures
PR: |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/reference/python-sdk.md`:
- Around line 90-92: The doc snippet assumes response.messages exists and calls
messages.append, which can throw if response.messages is None; update the
example around the variables response and messages (e.g., where
response.messages is read and before calling ai.call) to defensively initialize
messages = response.messages or [] (or use an explicit if/else) and then append
the new user message before passing messages into ai.call, ensuring the example
shows a safe, nil-checked pattern.
In `@holmes/core/toolset_manager.py`:
- Around line 380-387: The additional-toolset block currently may re-check
prerequisites already validated earlier; before building additional_to_check,
compute a set of names that have already been checked and exclude them (e.g.,
compare against the already-validated toolset names) so additional_to_check =
[ts for ts in all_toolsets_with_status if ts.name in {ats.name for ats in
self.additional_toolsets} and ts.enabled and ts.name not in
already_checked_names]; then call
self.check_toolset_prerequisites(additional_to_check) only if
additional_to_check is non-empty. Ensure you use the existing symbols
additional_toolsets, all_toolsets_with_status, additional_to_check and the
method check_toolset_prerequisites to locate and apply the change.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
docs/installation/python-installation.mddocs/reference/python-sdk.mdholmes/config.pyholmes/core/toolset_manager.py
mainred
reviewed
Feb 26, 2026
mainred
left a comment
mainred
left a comment
Collaborator
There was a problem hiding this comment.
Good to know we support customized python toolset.
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
holmes/core/toolset_manager.py (1)
367-385:⚠️ Potential issue | 🟠 MajorAvoid duplicate prerequisite checks for additional toolsets.
additional_to_checkcan include toolsets already checked earlier in this method, causing duplicate prerequisite executions (including duplicate external health calls).🔧 Suggested dedup patch
+ already_checked_names: set[str] = set() if using_cached: # Lazy initialization: only run fast config-validity checks on startup # (static flags and env vars). Callable and command prerequisites are # deferred until the first time the LLM uses a tool from the toolset. lazy_toolsets: List[Toolset] = [] eager_toolsets: List[Toolset] = [] @@ self._check_config_prerequisites(lazy_toolsets) if eager_toolsets: self.check_toolset_prerequisites(eager_toolsets) + already_checked_names.update(ts.name for ts in eager_toolsets) else: self.check_toolset_prerequisites(enabled_toolsets_from_cache) + already_checked_names.update( + ts.name for ts in enabled_toolsets_from_cache + ) @@ - # Additional Python toolsets passed programmatically are not cached, - # so always check their prerequisites. + # Additional Python toolsets should run full prerequisite checks on each load, + # but skip toolsets already checked in this execution path. if self.additional_toolsets: + additional_names = {ats.name for ats in self.additional_toolsets} additional_to_check = [ ts for ts in all_toolsets_with_status - if ts.name in {ats.name for ats in self.additional_toolsets} and ts.enabled + if ts.enabled + and ts.name in additional_names + and ts.name not in already_checked_names ] - self.check_toolset_prerequisites(additional_to_check) + if additional_to_check: + self.check_toolset_prerequisites(additional_to_check)Also applies to: 409-417
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@holmes/core/toolset_manager.py` around lines 367 - 385, The code is running duplicate prerequisite checks because toolsets in additional_to_check may already be in the lists passed to _check_config_prerequisites / check_toolset_prerequisites; deduplicate before invoking these functions by filtering additional_to_check against already-seen toolsets (use a stable unique key such as toolset.id or (toolset.name, toolset.type) to identify toolsets), then only call _check_config_prerequisites and check_toolset_prerequisites with the filtered lists (apply the same dedupe logic where eager_toolsets/lazy_toolsets are combined with additional_to_check in both the startup branch and the later branch around check_toolset_prerequisites).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@holmes/core/toolset_manager.py`:
- Around line 367-385: The code is running duplicate prerequisite checks because
toolsets in additional_to_check may already be in the lists passed to
_check_config_prerequisites / check_toolset_prerequisites; deduplicate before
invoking these functions by filtering additional_to_check against already-seen
toolsets (use a stable unique key such as toolset.id or (toolset.name,
toolset.type) to identify toolsets), then only call _check_config_prerequisites
and check_toolset_prerequisites with the filtered lists (apply the same dedupe
logic where eager_toolsets/lazy_toolsets are combined with additional_to_check
in both the startup branch and the later branch around
check_toolset_prerequisites).
aantn
enabled auto-merge (squash)
March 2, 2026 09:15
Track which toolset names were already checked by the cache and CLI paths, and only run check_toolset_prerequisites on additional toolsets that weren't covered. https://claude.ai/code/session_01RuoZoiEtUGjL4MnxDec572 Signed-off-by: Claude <noreply@anthropic.com>
…1:57540/git/HolmesGPT/holmesgpt into claude/update-python-sdk-docs-h8iPW
…1:57540/git/HolmesGPT/holmesgpt into claude/update-python-sdk-docs-h8iPW
moshemorad
approved these changes
Mar 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a new comprehensive Python SDK reference guide and refactors the Python installation documentation to better organize SDK usage information.
Key Changes
New
docs/reference/python-sdk.md: Complete Python SDK reference documentation including:Config,ToolCallingLLM,LLMResult, andToolCallResultclassesbuild_initial_ask_messages()Refactored
docs/installation/python-installation.md:gpt-4.1toanthropic/claude-sonnet-4-5-20250929Updated
docs/reference/.nav.yml: Added Python SDK reference to navigation menuUpdated
mkdocs.yml: Added Python SDK reference to the documentation structureImplementation Details
The new Python SDK reference provides:
The refactoring consolidates SDK documentation into a dedicated reference guide while keeping the installation page focused on setup instructions.
https://claude.ai/code/session_01RuoZoiEtUGjL4MnxDec572
Summary by CodeRabbit
New Features
Documentation