Skip to content

Remove workload health feature and related artifacts #1270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aantn wants to merge 3 commits into from
Closed

Remove workload health feature and related artifacts #1270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e1a1e06
Remove workload health endpoints and related artifacts
aantn Dec 30, 2025
d673f2c
Remove eval mapping file per request
aantn Dec 30, 2025
2d88e8d
Remove unused workload issue helpers
aantn Jan 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 1 addition & 98 deletions docs/reference/http-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# HolmesGPT API Reference

## Overview
The HolmesGPT API provides endpoints for automated investigations, workload health checks, and conversational troubleshooting. This document describes each endpoint, its purpose, request fields, and example usage.
The HolmesGPT API provides endpoints for automated investigations and conversational troubleshooting. This document describes each endpoint, its purpose, request fields, and example usage.

@coderabbitai coderabbitai Bot Dec 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

wc -l docs/reference/http-api.md

Repository: HolmesGPT/holmesgpt

Length of output: 93

🏁 Script executed:

head -50 docs/reference/http-api.md | cat -n

Repository: HolmesGPT/holmesgpt

Length of output: 1976

🏁 Script executed:

# Check for markdown formatting issues (headers followed by lists without blank lines)
rg -n '^#+\s|^\*\s|^-\s|^\d+\.\s' docs/reference/http-api.md | head -30

Repository: HolmesGPT/holmesgpt

Length of output: 1293

🏁 Script executed:

# Check for headers/bold text followed directly by lists without blank line
# Pattern: lines starting with # or ** followed by lines starting with - or *
python3 << 'EOF'
with open('docs/reference/http-api.md', 'r') as f:
    lines = f.readlines()

issues = []
for i in range(len(lines) - 1):
    current_line = lines[i].rstrip()
    next_line = lines[i + 1].rstrip()
    
    # Check if current line is a header or bold text
    is_header = current_line.startswith('#')
    is_bold = '**' in current_line
    
    # Check if next line is a list item (with content, not blank)
    is_list_item = next_line.strip().startswith('-') or next_line.strip().startswith('*') or (next_line.strip() and next_line.strip()[0].isdigit() and '.' in next_line)
    
    if (is_header or is_bold) and is_list_item and next_line.strip():
        issues.append((i + 1, current_line, i + 2, next_line))

if issues:
    print("Found markdown formatting violations (header/bold followed directly by list):")
    for line_num, current, next_num, next_content in issues:
        print(f"\nLine {line_num}: {current[:80]}")
        print(f"Line {next_num}: {next_content[:80]}")
else:
    print("No markdown formatting violations found (headers/bold properly separated from lists)")
EOF

Repository: HolmesGPT/holmesgpt

Length of output: 149

🏁 Script executed:

# Run the verification script from the review comment
# Verify no stale references to removed workload health endpoints

# Search documentation files for references to removed endpoints
echo "Checking documentation files..."
rg -n --type=md -C2 'workload_health_check|workload_health_chat' docs/

# Search Python code comments for references to removed endpoints
echo -e "\nChecking Python code comments..."
rg -n --type=py -C2 '/api/workload_health'

# Search for any remaining "workload health" or "workload_health" references in docs
echo -e "\nChecking for general workload health references in docs..."
rg -n --type=md 'workload.?health' docs/

Repository: HolmesGPT/holmesgpt

Length of output: 188

🏁 Script executed:

# Check for markdown formatting violations more efficiently
rg -n '^(#{1,6}\s|[*_]{2}[^*_]+[*_]{2})' docs/reference/http-api.md | head -40

Repository: HolmesGPT/holmesgpt

Length of output: 1589

🏁 Script executed:

# For each header or bold, check if the next non-empty line is a list
python3 << 'EOF'
with open('docs/reference/http-api.md', 'r') as f:
    lines = f.readlines()

violations = []
for i in range(len(lines)):
    current = lines[i].rstrip()
    # Check if it's a header
    if current.startswith('#'):
        # Look for next non-empty line
        j = i + 1
        while j < len(lines) and not lines[j].strip():
            j += 1
        
        if j < len(lines):
            next_line = lines[j].rstrip()
            # Check if next non-empty line is a list item
            stripped = next_line.strip()
            if stripped.startswith('-') or stripped.startswith('*') or (stripped and stripped[0].isdigit() and '.' in stripped.split()[0]):
                # There should be a blank line between header and list
                if i + 1 == j:  # No blank line
                    violations.append((i+1, current[:60], j+1, next_line[:60]))

if violations:
    print("Markdown formatting violations found:")
    for line_num, header, list_line, list_content in violations:
        print(f"Line {line_num}: {header}")
        print(f"  -> Line {list_line}: {list_content}")
        print()
else:
    print("✓ No violations found - all headers/bold text properly separated from lists")
EOF

Repository: HolmesGPT/holmesgpt

Length of output: 673

🏁 Script executed:

# More precise check for violations: headers/bold followed directly by actual list items
python3 << 'EOF'
import re

with open('docs/reference/http-api.md', 'r') as f:
    lines = f.readlines()

violations = []
for i in range(len(lines) - 1):
    current = lines[i].rstrip()
    next_line = lines[i + 1].rstrip()
    next_stripped = next_line.strip()
    
    # Check if current line is header or contains bold text
    is_header = current.startswith('#')
    has_bold = '**' in current and not current.startswith('#')
    
    # Check if NEXT line (immediately after) is a list item
    is_list = (next_stripped.startswith('-') or 
               next_stripped.startswith('*') or
               (next_stripped and re.match(r'^\d+\.', next_stripped)))
    
    if (is_header or has_bold) and is_list and next_stripped:
        violations.append((i+1, current[:70], i+2, next_line[:70]))

if violations:
    print("Found violations (header/bold directly followed by list):")
    for line_num, header, list_line, list_content in violations:
        print(f"\nLine {line_num}: {header}")
        print(f"Line {list_line}: {list_content}")
else:
    print("✓ No violations found")
EOF

Repository: HolmesGPT/holmesgpt

Length of output: 821

🏁 Script executed:

# Also check the specific area around line 4 mentioned in the review
sed -n '1,15p' docs/reference/http-api.md | cat -n

Repository: HolmesGPT/holmesgpt

Length of output: 737

Add blank lines between headers/bold text and lists to comply with MkDocs rendering requirements.

The file violates the markdown formatting guideline at multiple locations. Headers and bold text must be separated from lists by a blank line, but the following sections have direct adjacency:

  • Line 46-47: ### /api/chat (POST) → **Description:** (followed by content)
  • Line 85-86: ### /api/investigate (POST) → **Description:**
  • Line 145-146: ### /api/stream/investigate (POST) → **Description:**
  • Line 198-199: ### /api/issue_chat (POST) → **Description:**
  • Line 211-212: **investigation_result** object: → - result (string, optional)...
  • Line 248-249: ### /api/model (GET) → **Description:**

Add blank lines after each header and bold section header to ensure proper MkDocs rendering.

🤖 Prompt for AI Agents 
In docs/reference/http-api.md around the indicated lines (46-47, 85-86, 145-146,
198-199, 211-212, 248-249) the headers and bold section headers are immediately
followed by list content which breaks MkDocs rendering; fix by inserting a
single blank line after each affected header or bolded line so there is an empty
line between the header/bold text and the following list/content (i.e., add one
newline after those specific lines).


## Model Parameter Behavior

Expand Down Expand Up @@ -245,103 +245,6 @@ curl -X POST http://<HOLMES-URL>/api/issue_chat \

---

### `/api/workload_health_check` (POST)
**Description:** Performs a health check on a specified workload (e.g., a Kubernetes deployment).

#### Request Fields

| Field | Required | Default | Type | Description |
|-------------------------|----------|--------------------------------------------|-----------|--------------------------------------------------|
| ask | Yes | | string | User's question |
| resource | Yes | | object | Resource details (e.g., name, kind) |
| alert_history_since_hours| No | 24 | float | How many hours back to check alerts |
| alert_history | No | true | boolean | Whether to include alert history |
| stored_instructions | No | true | boolean | Use stored instructions |
| instructions | No | [] | list | Additional instructions |
| include_tool_calls | No | false | boolean | Include tool calls in response |
| include_tool_call_results| No | false | boolean | Include tool call results in response |
| prompt_template | No | "builtin://kubernetes_workload_ask.jinja2" | string | Prompt template to use |
| model | No | | string | Model name from your `modelList` configuration |

**Example**
```bash
curl -X POST http://<HOLMES-URL>/api/workload_health_check \
-H "Content-Type: application/json" \
-d '{
"ask": "Why is my deployment unhealthy?",
"resource": {"name": "my-deployment", "kind": "Deployment"},
"alert_history_since_hours": 12
}'
```

**Example** Response
```json
{
"analysis": "Deployment 'my-deployment' is unhealthy due to repeated CrashLoopBackOff events.",
"sections": null,
"tool_calls": [
{
"tool_call_id": "2",
"tool_name": "kubectl_get_events",
"description": "Fetch recent events",
"result": {"events": "..."}
}
],
"instructions": [...]
}
```

---

### `/api/workload_health_chat` (POST)
**Description:** Conversational interface for discussing the health of a workload.

#### Request Fields

| Field | Required | Default | Type | Description |
|-------------------------|----------|---------|-----------|--------------------------------------------------|
| ask | Yes | | string | User's question |
| workload_health_result | Yes | | object | Previous health check result (see below) |
| resource | Yes | | object | Resource details |
| conversation_history | No | | list | Conversation history (first message must be system)|
| model | No | | string | Model name from your `modelList` configuration |

**workload_health_result** object:
- `analysis` (string, optional): Previous analysis
- `tools` (list, optional): Tools used/results

**Example**
```bash
curl -X POST http://<HOLMES-URL>/api/workload_health_chat \
-H "Content-Type: application/json" \
-d '{
"ask": "Check the workload health.",
"workload_health_result": {
"analysis": "Previous health check: all good.",
"tools": []
},
"resource": {"name": "my-deployment", "kind": "Deployment"},
"conversation_history": [
{"role": "system", "content": "You are a helpful assistant."}
]
}'
```

**Example** Response
```json
{
"analysis": "The deployment 'my-deployment' is healthy. No recent issues detected.",
"conversation_history": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Check the workload health."},
{"role": "assistant", "content": "The deployment 'my-deployment' is healthy. No recent issues detected."}
],
"tool_calls": [...]
}
```

---

### `/api/model` (GET)
**Description:** Returns a list of available AI models that can be used for investigations and chat.

Expand Down
222 changes: 0 additions & 222 deletions holmes/core/conversations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
from holmes.core.models import (
ToolCallConversationResult,
IssueChatRequest,
WorkloadHealthChatRequest,
)
from holmes.plugins.prompts import load_and_render_prompt
from holmes.core.tool_calling_llm import ToolCallingLLM
Expand Down Expand Up @@ -425,224 +424,3 @@ def build_chat_messages(
)
truncate_tool_messages(conversation_history, tool_size) # type: ignore
return conversation_history # type: ignore


def build_workload_health_chat_messages(
workload_health_chat_request: WorkloadHealthChatRequest,
ai: ToolCallingLLM,
config: Config,
global_instructions: Optional[Instructions] = None,
runbooks: Optional[RunbookCatalog] = None,
):
"""
This function generates a list of messages for workload health conversation and ensures that the message sequence adheres to the model's context window limitations
by truncating tool outputs as necessary before sending to llm.

We always expect conversation_history to be passed in the openAI format which is supported by litellm and passed back by us.
That's why we assume that first message in the conversation is system message and truncate tools for it.

System prompt handling:
1. For new conversations (empty conversation_history):
- Creates a new system prompt using kubernetes_workload_chat.jinja2 template
- Includes workload analysis, tools (if any), and resource information
- If there are tools, calculates appropriate tool size and truncates tool outputs

2. For existing conversations:
- Preserves the conversation history
- Updates the first message (system prompt) with recalculated content
- Truncates tool outputs if necessary to fit context window
- Maintains the original conversation flow while ensuring context limits

Example structure of conversation history:
conversation_history = [
# System prompt with workload analysis
{"role": "system", "content": "...."},
# User message asking about workload health
{"role": "user", "content": "What's the current health status of my deployment?"},
# Assistant initiates a tool call
{
"role": "assistant",
"content": None,
"tool_call": {
"name": "check_workload_metrics",
"arguments": "{\"namespace\": \"default\", \"workload\": \"my-deployment\"}"
}
},
# Tool/Function response
{
"role": "tool",
"name": "check_workload_metrics",
"content": "{\"cpu_usage\": \"45%\", \"memory_usage\": \"60%\", \"status\": \"Running\"}"
},
# Assistant's final response to the user
{
"role": "assistant",
"content": "Your deployment is running normally with CPU usage at 45% and memory usage at 60%."
},
]
"""

template_path = "builtin://kubernetes_workload_chat.jinja2"

conversation_history = workload_health_chat_request.conversation_history
user_prompt = workload_health_chat_request.ask
workload_analysis = workload_health_chat_request.workload_health_result.analysis
tools_for_workload = workload_health_chat_request.workload_health_result.tools
resource = workload_health_chat_request.resource

if not conversation_history or len(conversation_history) == 0:
runbooks_ctx = generate_runbooks_args(
runbook_catalog=runbooks,
global_instructions=global_instructions,
)
user_prompt = generate_user_prompt(
user_prompt,
runbooks_ctx,
)

number_of_tools_for_workload = len(tools_for_workload) # type: ignore
if number_of_tools_for_workload == 0:
system_prompt = load_and_render_prompt(
template_path,
{
"workload_analysis": workload_analysis,
"tools_called_for_workload": tools_for_workload,
"resource": resource,
"toolsets": ai.tool_executor.toolsets,
"cluster_name": config.cluster_name,
"runbooks_enabled": True if runbooks else False,
},
)
messages = [
{
"role": "system",
"content": system_prompt,
},
{
"role": "user",
"content": user_prompt,
},
]
return messages

template_context_without_tools = {
"workload_analysis": workload_analysis,
"tools_called_for_workload": None,
"resource": resource,
"toolsets": ai.tool_executor.toolsets,
"cluster_name": config.cluster_name,
"runbooks_enabled": True if runbooks else False,
}
system_prompt_without_tools = load_and_render_prompt(
template_path, template_context_without_tools
)
messages_without_tools = [
{
"role": "system",
"content": system_prompt_without_tools,
},
{
"role": "user",
"content": user_prompt,
},
]
tool_size = calculate_tool_size(
ai, messages_without_tools, number_of_tools_for_workload
)

truncated_workload_result_tool_calls = [
ToolCallConversationResult(
name=tool.name,
description=tool.description,
output=tool.output[:tool_size],
)
for tool in tools_for_workload # type: ignore
]

truncated_template_context = {
"workload_analysis": workload_analysis,
"tools_called_for_workload": truncated_workload_result_tool_calls,
"resource": resource,
"toolsets": ai.tool_executor.toolsets,
"cluster_name": config.cluster_name,
"runbooks_enabled": True if runbooks else False,
}
system_prompt_with_truncated_tools = load_and_render_prompt(
template_path, truncated_template_context
)
return [
{
"role": "system",
"content": system_prompt_with_truncated_tools,
},
{
"role": "user",
"content": user_prompt,
},
]

runbooks_ctx = generate_runbooks_args(
runbook_catalog=runbooks,
global_instructions=global_instructions,
)
user_prompt = generate_user_prompt(
user_prompt,
runbooks_ctx,
)

conversation_history.append(
{
"role": "user",
"content": user_prompt,
}
)
number_of_tools = len(tools_for_workload) + len( # type: ignore
[message for message in conversation_history if message.get("role") == "tool"]
)

if number_of_tools == 0:
return conversation_history

conversation_history_without_tools = [
message for message in conversation_history if message.get("role") != "tool"
]
template_context_without_tools = {
"workload_analysis": workload_analysis,
"tools_called_for_workload": None,
"resource": resource,
"toolsets": ai.tool_executor.toolsets,
"cluster_name": config.cluster_name,
"runbooks_enabled": True if runbooks else False,
}
system_prompt_without_tools = load_and_render_prompt(
template_path, template_context_without_tools
)
conversation_history_without_tools[0]["content"] = system_prompt_without_tools

tool_size = calculate_tool_size(
ai, conversation_history_without_tools, number_of_tools
)

truncated_workload_result_tool_calls = [
ToolCallConversationResult(
name=tool.name, description=tool.description, output=tool.output[:tool_size]
)
for tool in tools_for_workload # type: ignore
]

template_context = {
"workload_analysis": workload_analysis,
"tools_called_for_workload": truncated_workload_result_tool_calls,
"resource": resource,
"toolsets": ai.tool_executor.toolsets,
"cluster_name": config.cluster_name,
"runbooks_enabled": True if runbooks else False,
}
system_prompt_with_truncated_tools = load_and_render_prompt(
template_path, template_context
)
conversation_history[0]["content"] = system_prompt_with_truncated_tools

truncate_tool_messages(conversation_history, tool_size)

return conversation_history
Loading