Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions .github/workflows/eval-regression.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,6 @@ jobs:

// Collect eval names from fixture directories
const askHolmesPath = 'tests/llm/fixtures/test_ask_holmes';
const investigatePath = 'tests/llm/fixtures/test_investigate';

const getEvalNames = (dir) => {
try {
Expand All @@ -106,12 +105,10 @@ jobs:
};

const askHolmesEvals = getEvalNames(askHolmesPath);
const investigateEvals = getEvalNames(investigatePath);

const body = `## 📋 Available Eval Names\n\n` +
`Use these with \`filter:\` in your \`/eval\` command.\n\n` +
`**test_ask_holmes:**\n${askHolmesEvals}\n\n` +
`**test_investigate:**\n${investigateEvals}`;
`**test_ask_holmes:**\n${askHolmesEvals}`;

await github.rest.issues.createComment({
owner: context.repo.owner,
Expand Down Expand Up @@ -661,7 +658,16 @@ jobs:
# Unset MODEL if empty so get_models() uses DEFAULT_MODEL
[[ -z "$MODEL" ]] && unset MODEL

PYTEST_ARGS=(--collect-only -q tests/llm/test_ask_holmes.py tests/llm/test_investigate.py -m "$EVAL_MARKER_EXPR")
# Collect test files dynamically
TEST_FILES=()
for f in tests/llm/test_ask_holmes.py tests/llm/test_holmes_checks.py; do
[[ -f "$f" ]] && TEST_FILES+=("$f")
done
if [[ ${#TEST_FILES[@]} -eq 0 ]]; then
echo "❌ No test files found"
exit 1
fi
PYTEST_ARGS=(--collect-only -q "${TEST_FILES[@]}" -m "$EVAL_MARKER_EXPR")
[[ -n "$EVAL_FILTER" ]] && PYTEST_ARGS+=(-k "$EVAL_FILTER")

# Collect test names
Expand Down Expand Up @@ -849,7 +855,16 @@ jobs:

START_TIME=$(date +%s)

PYTEST_ARGS=(--no-cov tests/llm/test_ask_holmes.py tests/llm/test_investigate.py -s -n10 -m "$EVAL_MARKER_EXPR")
# Collect test files dynamically
TEST_FILES=()
for f in tests/llm/test_ask_holmes.py tests/llm/test_holmes_checks.py; do
[[ -f "$f" ]] && TEST_FILES+=("$f")
done
if [[ ${#TEST_FILES[@]} -eq 0 ]]; then
echo "❌ No test files found"
exit 1
fi
PYTEST_ARGS=(--no-cov "${TEST_FILES[@]}" -s -n10 -m "$EVAL_MARKER_EXPR")
[[ -n "$EVAL_FILTER" ]] && PYTEST_ARGS+=(-k "$EVAL_FILTER")
poetry run pytest "${PYTEST_ARGS[@]}" || true

Expand Down
3 changes: 0 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@ check:
poetry run pre-commit run -a


test-llm-investigate:
poetry run pytest tests/llm/test_investigate.py -n 6 -vv

test-llm-ask-holmes:
poetry run pytest tests/llm/test_ask_holmes.py -n 6 -vv

Expand Down
224 changes: 9 additions & 215 deletions docs/reference/http-api.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# HolmesGPT API Reference

## Overview
The HolmesGPT API provides endpoints for automated investigations and conversational troubleshooting. This document describes each endpoint, its purpose, request fields, and example usage.
The HolmesGPT API provides endpoints for conversational troubleshooting. This document describes each endpoint, its purpose, request fields, and example usage.

## Model Parameter Behavior

Expand Down Expand Up @@ -229,174 +229,8 @@ For the most up-to-date list of vision-enabled models, see the [LiteLLM Vision D

---

### `/api/investigate` (POST)
**Description:** Initiate an automated investigation of an issue or incident.

#### Request Fields

| Field | Required | Default | Type | Description |
|-------------------------|----------|--------------------------------------------|-----------|--------------------------------------------------|
| source | Yes | | string | Source of the issue (e.g., "prometheus") |
| title | Yes | | string | Title of the investigation |
| description | Yes | | string | Description of the issue |
| subject | Yes | | object | Subject details (e.g., resource info) |
| context | Yes | | object | Additional context |
| source_instance_id | No | "ApiRequest" | string | Source instance identifier |
| include_tool_calls | No | false | boolean | Include tool calls in response |
| include_tool_call_results| No | false | boolean | Include tool call results in response |
| prompt_template | No | "builtin://generic_investigation.jinja2" | string | Prompt template to use |
| sections | No | | object | Structured output sections |
| model | No | | string | Model name from your `modelList` configuration |

**Example**
```bash
curl -X POST http://<HOLMES-URL>/api/investigate \
-H "Content-Type: application/json" \
-d '{
"source": "prometheus",
"title": "Pod CrashLoopBackOff",
"description": "Pod is crashing repeatedly",
"subject": {"namespace": "default", "pod": "my-pod"},
"context": {},
"include_tool_calls": true,
"model": "claude-sonnet"
}'
```

> **Note:** The `model` value must be a key name from your `modelList` configuration, not a direct model identifier.

**Example** Response
```json
{
"analysis": "The pod 'my-pod' in namespace 'default' is crashing due to an OOMKilled event. Consider increasing memory limits.",
"sections": {
"Alert Explanation": "...",
"Key Findings": "...",
"Conclusions and Possible Root Causes": "...",
"Next Steps": "...",
"App or Infra?": "...",
"External links": "..."
},
"tool_calls": [
{
"tool_call_id": "1",
"tool_name": "kubectl_logs",
"description": "Fetch pod logs",
"result": {"logs": "..."}
}
],
"instructions": [...]
}
```

---

### `/api/stream/investigate` (POST)
**Description:** Same as `/api/investigate`, but returns results as a stream for real-time updates.

#### Request Fields
Same as [`/api/investigate`](#apiinvestigate-post).

**Example**
```bash
curl -N -X POST http://<HOLMES-URL>/api/stream/investigate \
-H "Content-Type: application/json" \
-d '{
"source": "prometheus",
"title": "Pod CrashLoopBackOff",
"description": "Pod is crashing repeatedly",
"subject": {"namespace": "default", "pod": "my-pod"},
"context": {},
"include_tool_calls": true,
"model": "claude-sonnet"
}'
```

**Example** Response (streamed)
```bash
event: start_tool_calling
data: {"tool_name": "kubectl_describe", "id": "call_0"}

event: start_tool_calling
data: {"tool_name": "kubectl_logs", "id": "call_1"}

event: start_tool_calling
data: {"tool_name": "kubectl_previous_logs", "id": "call_2"}

event: start_tool_calling
data: {"tool_name": "kubectl_memory_requests_namespace", "id": "call_3"}

event: tool_calling_result
data: {"tool_call_id": "call_3", "role": "tool", "description": "kubectl get pods -n default -o ...", "name": "kubectl_memory_requests_namespace", "result": {...}}

event: tool_calling_result
data: {"tool_call_id": "call_0", "role": "tool", "description": "kubectl describe pod my-pod -n default", "name": "kubectl_describe", "result": {...}}

event: tool_calling_result
data: {"tool_call_id": "call_2", "role": "tool", "description": "kubectl logs my-pod -n default --previous", "name": "kubectl_previous_logs", "result": {...}}

event: tool_calling_result
data: {"tool_call_id": "call_1", "role": "tool", "description": "kubectl logs my-pod -n default", "name": "kubectl_logs", "result": {...}}

event: ai_answer_end
data: {"sections": {"Alert Explanation": ...}}
```

---

### `/api/issue_chat` (POST)
**Description:** Conversational interface for discussing a specific issue or incident, with context from a previous investigation.

#### Request Fields

| Field | Required | Default | Type | Description |
|-------------------------|----------|---------|-----------|--------------------------------------------------|
| ask | Yes | | string | User's question |
| investigation_result | Yes | | object | Previous investigation result (see below) |
| issue_type | Yes | | string | Type of issue |
| conversation_history | No | | list | Conversation history (first message must be system)|
| model | No | | string | Model name from your `modelList` configuration |

**investigation_result** object:
- `result` (string, optional): Previous analysis
- `tools` (list, optional): Tools used/results

**Example**
<!-- test: status=200, has_fields=analysis|conversation_history, id=issue_chat_basic -->
```bash
curl -X POST http://<HOLMES-URL>/api/issue_chat \
-H "Content-Type: application/json" \
-d '{
"ask": "How do I fix this issue?",
"investigation_result": {
"result": "Pod crashed due to OOM.",
"tools": []
},
"issue_type": "CrashLoopBackOff",
"conversation_history": [
{"role": "system", "content": "You are a helpful assistant."}
]
}'
```

**Example** Response
```json
{
"analysis": "To fix the CrashLoopBackOff, increase the memory limit for the pod and check for memory leaks in the application.",
"conversation_history": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "How do I fix this issue?"},
{"role": "assistant", "content": "To fix the CrashLoopBackOff, increase the memory limit for the pod and check for memory leaks in the application."}
],
"tool_calls": [...],
"follow_up_actions": [...]
}
```

---

### `/api/model` (GET)
**Description:** Returns a list of available AI models that can be used for investigations and chat.
**Description:** Returns a list of available AI models that can be used for chat.

**Example**
<!-- test: status=200, has_fields=model_name, id=model_list -->
Expand All @@ -415,7 +249,7 @@ curl http://<HOLMES-URL>/api/model

## Server-Sent Events (SSE) Reference

All streaming endpoints (`/api/stream/investigate`, `/api/stream/chat`, `/api/stream/issue_chat`, etc.) emit Server-Sent Events (SSE) to provide real-time updates during the investigation or chat process.
Streaming endpoints (e.g., `/api/chat` with `stream: true`) emit Server-Sent Events (SSE) to provide real-time updates during the chat process.

### Metadata Object Reference

Expand Down Expand Up @@ -578,26 +412,9 @@ Emitted when the AI has a text message or reasoning to share (typically before t

#### `ai_answer_end`

Emitted when the investigation or chat is complete. This is the final event in the stream.

**For RCA/Investigation (`/api/stream/investigate`):**
```json
{
"sections": {
"Alert Explanation": "...",
"Key Findings": "...",
"Conclusions and Possible Root Causes": "...",
"Next Steps": "...",
"App or Infra?": "...",
"External links": "..."
},
"analysis": "Full analysis text...",
"instructions": ["runbook1", "runbook2"],
"metadata": {...}
}
```
Emitted when the chat is complete. This is the final event in the stream.

**For Chat (`/api/stream/chat`, `/api/stream/issue_chat`):**
**Payload:**
```json
{
"analysis": "The issue can be resolved by...",
Expand All @@ -618,18 +435,9 @@ Emitted when the investigation or chat is complete. This is the final event in t
}
```

**Common Fields:**
**Fields:**

- `metadata` (object): See [Metadata Object Reference](#metadata-object-reference) for complete structure including token usage, truncations, and compaction info

**RCA-Specific Fields:**

- `sections` (object): Structured investigation output with predefined sections (customizable via request)
- `analysis` (string): Full analysis text (markdown format)
- `instructions` (array): List of runbooks that were used during investigation

**Chat-Specific Fields:**

- `analysis` (string): The AI's response (markdown format)
- `conversation_history` (array): Complete conversation history including the latest response
- `follow_up_actions` (array|null): Optional follow-up actions the user can take
Expand Down Expand Up @@ -688,7 +496,7 @@ To continue after approval, send a new request with `tool_decisions`:

#### `token_count`

Emitted periodically to provide token usage updates during the investigation. This event is sent after each LLM iteration to help track resource consumption in real-time.
Emitted periodically to provide token usage updates during the chat. This event is sent after each LLM iteration to help track resource consumption in real-time.

**Payload:**
```json
Expand All @@ -699,7 +507,7 @@ Emitted periodically to provide token usage updates during the investigation. Th

**Fields:**

- `metadata` (object): See [Metadata Object Reference](#metadata-object-reference) for complete token usage structure. This event provides the same metadata structure as other events, allowing you to monitor token consumption throughout the investigation
- `metadata` (object): See [Metadata Object Reference](#metadata-object-reference) for complete token usage structure. This event provides the same metadata structure as other events, allowing you to monitor token consumption throughout the chat

---

Expand Down Expand Up @@ -759,20 +567,6 @@ Emitted when an error occurs during processing.

## Event Flow Examples

### Typical RCA Investigation Flow

```
1. start_tool_calling (tool 1)
2. start_tool_calling (tool 2)
3. tool_calling_result (tool 1)
4. tool_calling_result (tool 2)
5. token_count
6. start_tool_calling (tool 3)
7. tool_calling_result (tool 3)
8. token_count
9. ai_answer_end
```

### Chat with Approval Flow

```
Expand All @@ -784,7 +578,7 @@ Emitted when an error occurs during processing.
6. approval_required
[Client sends approval decisions]
1. tool_calling_result (approved tool executed)
[investigation resumes]
[chat resumes]
```

### Chat with History Compaction
Expand Down
3 changes: 1 addition & 2 deletions docs/reference/python-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,8 +273,7 @@ Main configuration class (`holmes.config.Config`).
|--------|---------|-------------|
| `Config.load_from_file(config_file, **kwargs)` | `Config` | Load configuration from a YAML file. |
| `Config.load_from_env()` | `Config` | Load configuration from environment variables. |
| `create_console_toolcalling_llm()` | `ToolCallingLLM` | Create an AI instance for asking questions. |
| `create_console_issue_investigator()` | `IssueInvestigator` | Create an AI instance for investigating alerts. |
| `create_console_toolcalling_llm()` | `ToolCallingLLM` | Create an AI instance for asking questions and investigating alerts. |
| `get_runbook_catalog()` | `RunbookCatalog` or `None` | Get the loaded runbook catalog. |

### `ToolCallingLLM`
Expand Down
Loading
Loading