Add toolset to trigger intentional OOM kill

docs/development/stress-testing-oom-kill.md

@@ -0,0 +1,110 @@
+# Stress Testing Holmes with Intentional OOM Kills
+
+> ⚠️ **Never enable this toolset in production.** It allocates ~30 GB of RAM on the Holmes host and is intended only for controlled, non-production stress tests.
+
+This guide explains how to intentionally trigger OOM kills using the built-in OOM toolsets, how to enable them safely, and how to confirm they are available.
+
+## Available Toolsets
+
+Holmes ships with two disabled-by-default toolsets for inducing an OOM kill:
+
+- **`oom_kill` (Python)** – `trigger_oom_kill` allocates ~30 GB and sleeps for a configurable duration.
+- **`oom_kill_bash` (YAML/bash)** – `trigger_oom_kill_bash` does the same via a bash-executed Python snippet and applies `ulimit -v 2097152` (2 GiB virtual memory cap) before allocation to reduce blast radius.
+
+Both toolsets require the environment variable `ALLOW_HOLMES_OOMKILL_TOOLSET` to pass prerequisites and must be explicitly enabled in configuration. They are **disabled by default** and will not be loaded unless you opt in.
+
+## Enabling via Helm/ArgoCD (cluster install)
+
+Use the correct values path for your deployment method.
+
+=== "Robusta Helm Chart (Holmes as subchart)"
+
+1. **Set the env guard** (required):
+   ```bash
+   argocd app set <APP_NAME> \
+     --helm-set-string holmes.additionalEnvVars[0].name=ALLOW_HOLMES_OOMKILL_TOOLSET \
+     --helm-set-string holmes.additionalEnvVars[0].value=true
+   ```
+
+2. **Enable the toolsets**:
+   ```bash
+   argocd app set <APP_NAME> \
+     --helm-set-string holmes.toolsets.oom_kill.enabled=true \
+     --helm-set-string holmes.toolsets.oom_kill_bash.enabled=true
+   ```
+
+3. **Sync to apply**:
+   ```bash
+   argocd app sync <APP_NAME>
+   ```
+
+4. **Verify** (optional):
+   ```bash
+   kubectl -n <holmes-namespace> exec -it <holmes-pod> -- \
+     cat /app/custom_toolset.yaml
+   # Expect oom_kill and oom_kill_bash present and enabled
+   ```
+
+=== "Holmes Helm Chart (direct)"
+
+1. **Set the env guard** (required):
+   ```bash
+   argocd app set <APP_NAME> \
+     --helm-set-string additionalEnvVars[0].name=ALLOW_HOLMES_OOMKILL_TOOLSET \
+     --helm-set-string additionalEnvVars[0].value=true
+   ```
+
+2. **Enable the toolsets**:
+   ```bash
+   argocd app set <APP_NAME> \
+     --helm-set-string toolsets.oom_kill.enabled=true \
+     --helm-set-string toolsets.oom_kill_bash.enabled=true
+   ```
+
+3. **Sync to apply**:
+   ```bash
+   argocd app sync <APP_NAME>
+   ```
+
+4. **Verify** (optional):
+   ```bash
+   kubectl -n <holmes-namespace> exec -it <holmes-pod> -- \
+     cat /app/custom_toolset.yaml
+   # Expect oom_kill and oom_kill_bash present and enabled
+   ```
+
+## Enabling in Local CLI Mode
+
+Add to your local config (e.g., `config.yaml`) and set the env guard before running the CLI:
+
+```yaml
+toolsets:
+  oom_kill:
+    enabled: true
+  oom_kill_bash:
+    enabled: true
+```
+
+Then run:
+```bash
+export ALLOW_HOLMES_OOMKILL_TOOLSET=true
+holmes --config ./config.yaml ...
+```
+
+Because both toolsets are disabled by default and gated by `ALLOW_HOLMES_OOMKILL_TOOLSET`, they will **not** be auto-enabled in local mode unless you explicitly enable them and set the env variable.
+
+## Using the Tools
+
+- **Python toolset**: `trigger_oom_kill` (param: `hold_seconds`, default 300).
+- **Bash toolset**: `trigger_oom_kill_bash` (param: `hold_seconds`, default 300).
+
+Example invocation (conceptual):
+```
+trigger_oom_kill: allocate ~30GB and sleep for 120s
+```
+
+## Safety Considerations
+
+- Keep this toolset out of production environments.
+- Ensure hosts have proper isolation; the process is expected to be OOM-killed.
+- Consider running in a dedicated test cluster or namespace.

holmes/core/tools.py

@@ -5,6 +5,7 @@
 import shlex
 import subprocess
 import tempfile
+import time
 from abc import ABC, abstractmethod
 from datetime import datetime
 from enum import Enum
@@ -30,21 +31,21 @@
     PrivateAttr,
 )
 from rich.console import Console
+from rich.table import Table
 
 from holmes.core.llm import LLM
 from holmes.core.openai_formatting import format_tool_to_open_ai_standard
-from holmes.plugins.prompts import load_and_render_prompt
 from holmes.core.transformers import (
     registry,
     TransformerError,
     Transformer,
 )
+from holmes.plugins.prompts import load_and_render_prompt
+from holmes.utils.config_utils import merge_transformers
+from holmes.utils.memory_limit import get_ulimit_prefix, check_oom_and_append_hint
 
 if TYPE_CHECKING:
     from holmes.core.transformers import BaseTransformer
-from holmes.utils.config_utils import merge_transformers
-import time
-from rich.table import Table
 
 logger = logging.getLogger(__name__)
 
@@ -497,8 +498,9 @@ def __invoke_script(self, params) -> str:
     def __execute_subprocess(self, cmd) -> Tuple[str, int]:
         try:
             logger.debug(f"Running `{cmd}`")
+            protected_cmd = get_ulimit_prefix() + cmd
             result = subprocess.run(
-                cmd,
+                protected_cmd,
                 shell=True,
                 text=True,
                 check=False,  # do not throw error, we just return the error code
@@ -507,7 +509,9 @@ def __execute_subprocess(self, cmd) -> Tuple[str, int]:
                 stderr=subprocess.STDOUT,
             )
 
-            return result.stdout.strip(), result.returncode
+            output = result.stdout.strip()
+            output = check_oom_and_append_hint(output, result.returncode)
+            return output, result.returncode
         except Exception as e:
             logger.error(
                 f"An unexpected error occurred while running '{cmd}': {e}",

holmes/plugins/toolsets/__init__.py

@@ -35,6 +35,7 @@
 from holmes.plugins.toolsets.internet.notion import NotionToolset
 from holmes.plugins.toolsets.kafka import KafkaToolset
 from holmes.plugins.toolsets.kubernetes_logs import KubernetesLogsToolset
+from holmes.plugins.toolsets.oom_kill import OOMKillToolset
 from holmes.plugins.toolsets.mcp.toolset_mcp import RemoteMCPToolset
 from holmes.plugins.toolsets.newrelic.newrelic import NewRelicToolset
 from holmes.plugins.toolsets.opensearch.opensearch import OpenSearchToolset
@@ -96,6 +97,7 @@ def load_python_toolsets(
         OpenSearchLogsToolset(),
         OpenSearchTracesToolset(),
         OpenSearchQueryAssistToolset(),
+        OOMKillToolset(),
         CoralogixToolset(),
         RabbitMQToolset(),
         GitToolset(),

holmes/plugins/toolsets/bash/common/bash.py

@@ -1,11 +1,14 @@
 import subprocess
+
 from holmes.core.tools import StructuredToolResult, StructuredToolResultStatus
+from holmes.utils.memory_limit import get_ulimit_prefix, check_oom_and_append_hint
 
 
 def execute_bash_command(cmd: str, timeout: int, params: dict) -> StructuredToolResult:
     try:
+        protected_cmd = get_ulimit_prefix() + cmd
         process = subprocess.run(
-            cmd,
+            protected_cmd,
             shell=True,
             executable="/bin/bash",
             stdout=subprocess.PIPE,
@@ -16,7 +19,8 @@ def execute_bash_command(cmd: str, timeout: int, params: dict) -> StructuredTool
         )
 
         stdout = process.stdout.strip() if process.stdout else ""
-        result_data = f"{cmd}\n" f"{stdout}"
+        stdout = check_oom_and_append_hint(stdout, process.returncode)
+        result_data = f"{cmd}\n{stdout}"
 
         if process.returncode == 0:
             status = (
@@ -44,10 +48,10 @@ def execute_bash_command(cmd: str, timeout: int, params: dict) -> StructuredTool
             params=params,
         )
     except FileNotFoundError:
-        # This might occur if /bin/bash is not found, or if shell=False and command is not found
+        # This might occur if /bin/bash is not found, or command is not found
         return StructuredToolResult(
             status=StructuredToolResultStatus.ERROR,
-            error="Error: Bash executable or command not found. Ensure bash is installed and the command is valid.",
+            error="Error: Bash executable or command not found.",
             params=params,
         )
     except Exception as e:

holmes/plugins/toolsets/oom_kill.py

@@ -0,0 +1,91 @@
+import time
+from typing import Any, Dict
+
+from holmes.core.tools import (
+    CallablePrerequisite,
+    StructuredToolResult,
+    StructuredToolResultStatus,
+    Tool,
+    ToolInvokeContext,
+    ToolParameter,
+    Toolset,
+    ToolsetTag,
+    ToolsetEnvironmentPrerequisite,
+)
+
+
+class TriggerOOMKill(Tool):
+    toolset: "OOMKillToolset"
+
+    def __init__(self, toolset: "OOMKillToolset"):
+        super().__init__(
+            name="trigger_oom_kill",
+            description=(
+                "Allocates approximately 30GB of memory on the Holmes host to provoke the "
+                "OOM killer. This is intended for stress testing only and will likely crash "
+                "the running process. No confirmation is required because this is meant for "
+                "automated stress scenarios."
+            ),
+            parameters={
+                "hold_seconds": ToolParameter(
+                    description=(
+                        "How long to keep the memory allocated before exiting. Defaults to 300 seconds."
+                    ),
+                    type="integer",
+                    required=False,
+                ),
+            },
+            toolset=toolset,  # type: ignore[call-arg]
+        )
+
+    def _invoke(self, params: dict, context: ToolInvokeContext) -> StructuredToolResult:
+        hold_seconds = params.get("hold_seconds", 300)
+        if not isinstance(hold_seconds, int) or hold_seconds <= 0:
+            return StructuredToolResult(
+                status=StructuredToolResultStatus.ERROR,
+                error="hold_seconds must be a positive integer.",
+                params=params,
+            )
+
+        size_bytes = 30 * 1024 * 1024 * 1024
+        print(
+            f"Allocating {{size_bytes / 1024 / 1024 / 1024:.0f}} GB of memory to intentionally trigger OOM kill; sleeping for {hold_seconds}s"
+        )
+        data = bytearray(size_bytes)  # type: ignore
+        time.sleep({hold_seconds})
+
+    def get_parameterized_one_liner(self, params: Dict[str, Any]) -> str:
+        hold_seconds = params.get("hold_seconds", 300)
+        return (
+            "python - <<'PY' ... # allocates ~30GB and sleeps for "
+            f"{hold_seconds}s to trigger OOM"
+        )
+
+
+class OOMKillToolset(Toolset):
+    def __init__(self):
+        super().__init__(
+            name="oom_kill",
+            enabled=False,
+            description=(
+                "Dangerous toolset that intentionally exhausts memory on the Holmes host to trigger an OOM kill. "
+                "Use only in controlled stress tests."
+            ),
+            docs_url=None,
+            icon_url=None,
+            prerequisites=[
+                ToolsetEnvironmentPrerequisite(env=["ALLOW_HOLMES_OOMKILL_TOOLSET"]),
+                CallablePrerequisite(callable=self.prerequisites_callable),
+            ],
+            tools=[TriggerOOMKill(self)],
+            experimental=True,
+            tags=[ToolsetTag.CORE],
+            is_default=False,
+        )
+
+    def prerequisites_callable(self, config: dict[str, Any]) -> tuple[bool, str]:
+        # No special configuration is required for this toolset.
+        return True, ""
+
+    def get_example_config(self) -> Dict[str, Any]:
+        return {}

holmes/plugins/toolsets/oom_kill.yaml

@@ -0,0 +1,28 @@
+toolsets:
+  oom_kill_bash:
+    description: "DANGEROUS: intentionally allocate ~30GB on the Holmes host to trigger OOM kill for stress testing (no confirmation required)."
+    docs_url: ""
+    icon_url: ""
+    tags:
+      - core
+    prerequisites:
+      - env:
+          - ALLOW_HOLMES_OOMKILL_TOOLSET
+    additional_instructions: |
+      ⚠️ This toolset is intentionally destructive. Only use in controlled environments.
+    tools:
+      - name: "trigger_oom_kill_bash"
+        description: "Allocate ~30GB of memory and hold it for a period to provoke the OOM killer. No confirmation required; intended for automated stress tests."
+        command: |
+          python - <<'PY'
+          import time
+
+          hold_seconds = int("{{ hold_seconds|default(300) }}")
+          if hold_seconds <= 0:
+              raise SystemExit("hold_seconds must be positive.")
+
+          size_bytes = 30 * 1024 * 1024 * 1024
+          print(f"Allocating {size_bytes / 1024 / 1024 / 1024:.0f} GB of memory to intentionally trigger OOM kill; sleeping for {hold_seconds}s")
+          data = bytearray(size_bytes)
+          time.sleep(hold_seconds)
+          PY