From b79402318648c433a131f99725a47c356b45edeb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 6 Mar 2026 06:50:06 +0000 Subject: [PATCH 1/4] Remove non-existent operator/troubleshooting.md from mkdocs nav The nav config referenced operator/troubleshooting.md which was never created, causing a MkDocs build warning. General troubleshooting content already exists at reference/troubleshooting.md. https://claude.ai/code/session_01M6qUqTTmhE9kGYkmmaAU3b Signed-off-by: Claude --- mkdocs.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 4a97d5bb04..53bd4397ae 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -32,7 +32,6 @@ nav: - "Health Checks": operator/health-checks.md - "Scheduled Health Checks": operator/scheduled-health-checks.md - "Configuration": operator/configuration.md - - "Troubleshooting": operator/troubleshooting.md - "Development Guide": operator/development.md - "AI Providers": - "Overview": ai-providers/index.md From f9d8079a546bb22988f7b7d611dd0453811ba5b0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 6 Mar 2026 07:52:25 +0000 Subject: [PATCH 2/4] Add remaining orphan pages to nav or exclude list - Add adr/operator-initial-architecture.md to Operator nav section - Exclude data-sources/permissions.md (redirect-only page) from docs https://claude.ai/code/session_01M6qUqTTmhE9kGYkmmaAU3b Signed-off-by: Claude --- mkdocs.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index f7c4be03ff..fd7e9bd7fe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,6 +9,7 @@ exclude_docs: | snippets/ README.md cncf-self-assesment.md + data-sources/permissions.md nav: - Home: index.md @@ -32,6 +33,7 @@ nav: - "Health Checks": operator/health-checks.md - "Scheduled Health Checks": operator/scheduled-health-checks.md - "Configuration": operator/configuration.md + - "Architecture": adr/operator-initial-architecture.md - "Development Guide": operator/development.md - "AI Providers": - "Overview": ai-providers/index.md From 9bb8a37e2d3169a69de059244987eda7e6066416 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 6 Mar 2026 09:34:44 +0000 Subject: [PATCH 3/4] Remove stale ADR and CNCF self-assessment docs - Delete adr/operator-initial-architecture.md (redundant with operator/ docs which are more complete and up to date) - Delete cncf-self-assesment.md (no longer needed) - Remove references to deleted files from nav, operator/index.md, and operator/development.md https://claude.ai/code/session_01M6qUqTTmhE9kGYkmmaAU3b Signed-off-by: Claude --- docs/adr/operator-initial-architecture.md | 446 ---------------------- docs/cncf-self-assesment.md | 135 ------- docs/operator/development.md | 1 - docs/operator/index.md | 2 +- mkdocs.yml | 3 - 5 files changed, 1 insertion(+), 586 deletions(-) delete mode 100644 docs/adr/operator-initial-architecture.md delete mode 100644 docs/cncf-self-assesment.md diff --git a/docs/adr/operator-initial-architecture.md b/docs/adr/operator-initial-architecture.md deleted file mode 100644 index f1b0381cd7..0000000000 --- a/docs/adr/operator-initial-architecture.md +++ /dev/null @@ -1,446 +0,0 @@ -# Holmes Operator Architecture - -## Overview - -The Holmes Operator extends Holmes with Kubernetes-native health check capabilities using Custom Resource Definitions (CRDs). Following the Kubernetes Job/CronJob pattern, it provides two CRD types: - -- **HealthCheck**: One-time execution checks that run immediately when created -- **ScheduledHealthCheck**: Recurring checks that create HealthCheck resources on a cron schedule - -The architecture maintains separation of concerns with a lightweight operator handling orchestration while stateless Holmes API servers perform the actual check execution. - -## Architecture Components - -### 1. Holmes Operator - -A lightweight Kubernetes controller built with [kopf](https://kopf.readthedocs.io/) that: - -- Watches HealthCheck CRDs -- Manages check scheduling using APScheduler -- Makes HTTP calls to Holmes API servers for check execution -- Updates CRD status with results -- Handles retry logic and error recovery - -### 2. Holmes API Servers - -Stateless FastAPI servers that: - -- Execute health checks via `/api/check/execute` endpoint -- Reuse existing CheckRunner implementation -- Scale horizontally based on load -- Share LLM rate limits and resource pools - -### 3. HealthCheck CRD - -One-time execution resource that: - -- Runs immediately upon creation -- Stores execution results in status -- Can be re-run via annotation -- Maintains audit trail of checks - -### 4. ScheduledHealthCheck CRD - -Recurring check resource that: - -- Defines cron schedule for checks -- Creates HealthCheck resources at scheduled times -- Tracks execution history -- Manages concurrency and history limits - -## Architecture Diagram - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Kubernetes Cluster │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────────────────────────┐ │ -│ │ Holmes Operator (kopf) │ │ -│ │ - CRD Watching │ │ -│ │ - Scheduling (APScheduler) │ │ -│ │ - Status Management │ │ -│ │ - Retry Logic │ │ -│ └──────┬───────────┬──────────┘ │ -│ │ │ │ -│ │ │ Watches/Updates │ -│ │ │ │ -│ │ ▼ │ -│ │ ┌────────────────────────────┐ │ -│ │ │ HealthCheck CRDs │ │ -│ │ │ (One-time execution) │ │ -│ │ └────────────────────────────┘ │ -│ │ │ │ -│ │ ▼ │ -│ │ ┌────────────────────────────┐ │ -│ │ │ ScheduledHealthCheck CRDs │ │ -│ │ │ (Recurring checks) │ │ -│ │ └────────────────────────────┘ │ -│ │ │ -│ │ HTTP API Calls │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────────────────────────┐ │ -│ │ Service: holmes-api │ │ -│ ├─────────────────────────────────────────┤ │ -│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │ │ server.py│ │ server.py│ │ server.py│ (Stateless)│ -│ │ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │ -│ │ └──────────┘ └──────────┘ └──────────┘ │ -│ └─────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## CRD Specification - -### HealthCheck Resource (One-time) - -```yaml -apiVersion: holmesgpt.dev/v1alpha1 -kind: HealthCheck -metadata: - name: frontend-health-check - namespace: default - labels: - # Set by ScheduledHealthCheck if created by schedule - holmesgpt.dev/scheduled-by: frontend-schedule -spec: - # Check definition - query: "Are all pods in deployment 'frontend' healthy with at least 2 ready replicas?" - timeout: 30 # seconds - mode: alert # or "monitor" (no alerts) - destinations: - - type: slack - config: - channel: "#alerts" - -status: - # Execution phase - phase: Completed # Pending/Running/Completed - startTime: "2024-01-01T00:00:00Z" - completionTime: "2024-01-01T00:00:02Z" - - # Result - result: pass # pass/fail/error - message: "All pods healthy with 3/3 replicas ready" - rationale: "Deployment 'frontend' has 3 ready replicas out of 3 desired." - duration: 2.0 - - # Conditions - conditions: - - type: Complete - status: "True" - lastTransitionTime: "2024-01-01T00:00:02Z" - reason: Pass - message: "Check completed successfully" -``` - -### ScheduledHealthCheck Resource (Recurring) - -```yaml -apiVersion: holmesgpt.dev/v1alpha1 -kind: ScheduledHealthCheck -metadata: - name: frontend-schedule - namespace: default -spec: - # Schedule - schedule: "*/5 * * * *" # Cron format (every 5 minutes) - query: "Are all pods in deployment 'frontend' healthy with at least 2 ready replicas?" - timeout: 30 - mode: alert - destinations: - - type: slack - config: - channel: "#alerts" - - # Schedule control - enabled: true # Enable or disable the schedule - # Note: The operator automatically: - # - Prevents concurrent runs (max_instances=1) - # - Maintains last 10 history items (hardcoded limit) - # - Coalesces missed jobs (if schedule was missed while disabled) - -status: - # Last execution - lastScheduleTime: "2024-01-01T00:05:00Z" - lastSuccessfulTime: "2024-01-01T00:05:00Z" - lastResult: pass - message: "Check completed successfully" - - # Active checks - active: - - name: frontend-schedule-20240101-000500-abc123 - namespace: default - uid: "12345-67890" - - # History (references to created HealthChecks) - history: - - executionTime: "2024-01-01T00:05:00Z" - result: pass - duration: 2.5 - checkName: frontend-schedule-20240101-000500-abc123 - - executionTime: "2024-01-01T00:00:00Z" - result: pass - duration: 2.3 - checkName: frontend-schedule-20240101-000000-def456 -``` - -## API Integration - -### Check Execution Endpoint - -The operator calls the Holmes API server to execute checks: - -```http -POST /api/check/execute -Content-Type: application/json -X-Check-Name: default/frontend-health - -{ - "query": "Are all pods in deployment 'frontend' healthy?", - "timeout": 30, - "mode": "alert", - "destinations": [ - { - "type": "slack", - "config": { - "channel": "#alerts" - } - } - ] -} -``` - -Response: -```json -{ - "status": "pass", - "message": "All pods are healthy with 3/3 replicas ready", - "duration": 2.5, - "rationale": "Deployment 'frontend' has 3 ready replicas out of 3 desired.", - "error": null -} -``` - -## Deployment - -### Operator Deployment - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: holmes-operator - namespace: holmes -spec: - replicas: 1 - selector: - matchLabels: - app: holmes-operator - template: - metadata: - labels: - app: holmes-operator - spec: - serviceAccountName: holmes-operator - containers: - - name: operator - image: robusta/holmes-operator:latest - command: ["kopf", "run", "-A", "--standalone", "/app/operator.py"] - env: - - name: HOLMES_API_URL - value: "http://holmes-api:8080" - - name: LOG_LEVEL - value: "INFO" - resources: - requests: - memory: "256Mi" - cpu: "100m" - limits: - memory: "512Mi" - cpu: "500m" -``` - -### RBAC Configuration - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: holmes-operator - namespace: holmes ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: holmes-operator -rules: -- apiGroups: ["holmesgpt.dev"] - resources: ["healthchecks"] - verbs: ["get", "list", "watch", "patch", "update"] -- apiGroups: [""] - resources: ["events"] - verbs: ["create"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: holmes-operator -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: holmes-operator -subjects: -- kind: ServiceAccount - name: holmes-operator - namespace: holmes -``` - -## Key Design Decisions - -### 1. Job/CronJob Pattern - -- **Decision**: Separate HealthCheck and ScheduledHealthCheck CRDs -- **Rationale**: Clear semantics, immediate feedback for one-time checks, follows K8s patterns -- **Trade-off**: Two CRDs instead of one, but clearer user experience - -### 2. Distributed Architecture - -- **Decision**: Separate operator from API servers -- **Rationale**: Better scalability, fault isolation, and resource efficiency -- **Trade-off**: Slightly more complex deployment - -### 3. HTTP Communication - -- **Decision**: Operator calls API via HTTP instead of direct execution -- **Rationale**: Reuses existing API infrastructure, enables horizontal scaling -- **Trade-off**: Network latency, requires service discovery - -### 4. APScheduler for Scheduling - -- **Decision**: Use APScheduler library instead of Kubernetes CronJobs -- **Rationale**: More efficient for frequent checks, better resource pooling -- **Trade-off**: Scheduling logic in operator instead of native K8s - -### 5. HealthCheck Resources as History - -- **Decision**: ScheduledHealthCheck creates HealthCheck resources -- **Rationale**: Natural audit trail, reusable checks, clear execution records -- **Trade-off**: More resources created, requires cleanup strategy - -## Monitoring and Observability - -### Metrics - -The operator exposes Prometheus metrics on port 8080 at `/metrics`: - -- `holmes_checks_scheduled_total` - Total number of scheduled checks (labels: namespace, name) -- `holmes_checks_executed_total` - Total number of executed checks (labels: namespace, name, type) -- `holmes_checks_failed_total` - Total number of failed checks (labels: namespace, name, type) -- `holmes_check_duration_seconds` - Check execution duration histogram (labels: namespace, name, type) -- `holmes_scheduled_checks_active` - Number of currently active scheduled checks (gauge) - -### Logging - -- Operator logs: Scheduling, CRD events, API calls -- API server logs: Check execution, tool calls, errors - -### kubectl Integration -```bash -# List one-time checks -kubectl get healthcheck - -# List scheduled checks -kubectl get scheduledhealthcheck - -# View check execution -kubectl describe healthcheck frontend-health-check - -# View schedule status -kubectl describe scheduledhealthcheck frontend-schedule - -# Get checks created by a schedule -kubectl get healthcheck -l holmesgpt.dev/scheduled-by=frontend-schedule - -# Disable a schedule -kubectl patch scheduledhealthcheck frontend-schedule --type='merge' -p '{"spec":{"enabled":false}}' -``` - -## Security Considerations - -### Authentication - -- Operator uses Kubernetes service account for API authentication -- API servers validate requests from operator -- Secrets for destinations stored in Kubernetes secrets - -### Network Policies - -- Restrict operator to API server communication -- Limit API server egress for tool execution - -### Resource Limits - -- Operator has strict resource limits -- Timeout enforcement for check execution (configurable, max 300s) - -### Known Limitations - -- **No built-in rate limiting**: The operator does not limit the rate of check creation or execution. Consider using ResourceQuotas or ValidatingAdmissionWebhooks to prevent DoS -- **No query validation**: Check queries are passed directly to the LLM without content validation. Ensure trusted users only -- **Cluster-wide RBAC**: The operator has cluster-wide access to HealthCheck CRDs - -## Future Enhancements - -### Phase 1 (MVP) - -- [x] Basic CRD and operator -- [x] Check execution via API -- [x] Status updates -- [x] Cron scheduling - -### Phase 2 (Future Enhancements) - -- [ ] High availability with leader election -- [ ] Check dependencies -- [ ] Webhook validation -- [ ] Result persistence to external storage -- [ ] Configurable concurrency policies -- [ ] Configurable history retention limits - -### Phase 3 - -- [ ] Multi-tenancy support -- [ ] Check templates -- [ ] Grafana dashboard integration -- [ ] Cost tracking per check - - -## Troubleshooting - -### Common Issues - -1. **Check not executing** - - - Verify operator is running: `kubectl logs -l app=holmes-operator` - - Check CRD status: `kubectl get healthcheck -o yaml` - - Ensure API service is accessible - -2. **Authentication errors** - - Verify service account permissions - - Check RBAC configuration - - Validate API server logs - -3. **Scheduling issues** - - - Check operator logs for scheduler errors - - Verify cron expression syntax - - Check for suspended status - -## References - -- [Kopf Documentation](https://kopf.readthedocs.io/) -- [APScheduler Documentation](https://apscheduler.readthedocs.io/) -- [Kubernetes CRD Best Practices](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -- [Holmes Check Documentation](./health-checks.md) diff --git a/docs/cncf-self-assesment.md b/docs/cncf-self-assesment.md deleted file mode 100644 index 8c5bff10fe..0000000000 --- a/docs/cncf-self-assesment.md +++ /dev/null @@ -1,135 +0,0 @@ - -# General Technical Review - HolmesGPT / Sandbox - -- **Project:** HolmesGPT -- **Project Version:** 0.11.5 -- **Website:** https://github.com/robusta-dev/holmesgpt -- **Date Updated:** 2025-07-16 -- **Template Version:** v1.0 -- **Description:** HolmesGPT is an AI agent that automates cloud-native troubleshooting, bridging knowledge gaps by investigating alerts, executing runbooks, and correlating observability data in cloud-native platforms. - -## Day 0 - Planning Phase - -### Scope -**Describe the roadmap process, how scope is determined for mid to long term features, as well as how the roadmap maps back to current contributions and maintainer ladder?** - -HolmesGPT follows an open and community-driven roadmap process. The roadmap is maintained publicly (via GitHub Projects and issues) and is shaped by a combination of the following inputs: - -* Community feedback from users and contributors, gathered through GitHub issues, Slack, and regular discussions -* Strategic alignment with the project’s long-term mission: enabling AI-powered root cause analysis in cloud-native environments -* Technical priorities based on gaps surfaced during usage or contributor pain points -* Upstream integration plans with core CNCF projects like Prometheus, Kubernetes, and OpenTelemetry - -Mid to long-term scope is defined quarterly and iterated collaboratively among maintainers - -**Describe the target persona or user(s) for the project?** -DevOps, SREs, and platform engineers - -**Explain the primary use case for the project. What additional use cases are supported by the project?** - -Two primary use cases: -1. Root-cause analysis of alerts (i.e. Prometheus alerts) -2. Troubleshooting problems (via free-text questions) in cloud-native environments - -In both cases, the analysis can be 100% autonomous, or driven by predefined runbooks. - -**Explain which use cases have been identified as unsupported by the project.** -General purpose AI agent use cases (outside of troubleshooting and RCA), coding agents, and security use cases. - -**Describe the intended types of organizations who would benefit from adopting this project. (i.e. financial services, any software manufacturer, organizations providing platform engineering services)?** - -End users running cloud-native services at scale, especially in larger environments with many microservices and potential for complex cascading failures. - -**Please describe any completed end user research and link to any reports.** -N/A - -### Usability - -**How should the target personas interact with your project?** -The most common entry point is via a cli tool that is run on-demand to troubleshoot a problem or an alert. - -**Describe the user experience (UX) and user interface (UI) of the project.** -The open source includes a CLI tool with an interactive console and /slash commands. Several vendors have built graphical user interfaces on top of that. - -**Describe how this project integrates with other projects in a production environment.** -The project exposes an HTTP API and a Helm chart for running the HTTP server in Kubernetes clusters. - -### Design - -**Explain the design principles and best practices the project is following.** -* Human-in-the-loop: users are able to interact with and guide HolmesGPT investigations -* Safety-first - The agent is restricted by default and only allowed to run safe commands. -* Interoperable – works seamlessly with existing observability stacks -* Kubernetes-native: Works with Prometheus, Loki, and other CNCF stack components. -* Extensible: Modular plugin system for adding new data sources, including external MCP servers - -**Outline or link to the project’s architecture requirements? Describe how they differ for Proof of Concept, Development, Test and Production environments, as applicable.** -Lightweight, can run as a standalone local CLI or in-cluster as an HTTP server. Typical during POC users start with the local CLI and when rolling into production use a more advanced setup. - -**Define any specific service dependencies the project relies on in the cluster.** -No relevant. - -**Describe how the project implements Identity and Access Management.** -HolmesGPT runs with user-provided credentials (e.g. service account) and respects whichever permissions were given to it. - -**Describe how the project has addressed sovereignty.** -HolmesGPT runs fully within the user’s infrastructure. All data—logs, metrics, traces, and AI-generated insights—remains under user control. There’s no dependency on external SaaS or third-party APIs unless explicitly configured. This ensures data privacy, compliance, and operational sovereignty. - -Regarding data sent to the LLM, here too users have the choice of providing their own LLM or using a trusted cloud provider of their choice. - -**Describe any compliance requirements addressed by the project.** -N/A - -**Describe the project’s High Availability requirements.** -Each request to HolmesGPT is stateless, so it's possible to run multiple instances. - -**Describe the project’s resource requirements, including CPU, Network and Memory.** -Minimal, similar to any standard Python application running in a Kubernetes cluster. We recommend some defaults in the Helm chart, but this can be customized by the user. - -**Describe the project’s storage requirements, including its use of ephemeral and/or persistent storage.** -N/A - -**Please outline the project’s API Design** -The project itself exposes a REST API, following standard conventions. We strive to maintain backwards compatibility, and to add new endpoints when changing something instead of breaking an existing endpoint. - -It will perform HTTP calls to collect data when investigating problems - the exact calls depend on which data sources the user enabled. - -We bump the major release number only on breaking changes. Minor releases are done about monthly, when there are substantial new features. Bug fixes are done as needed with a patch release. - -**Describe how the project is installed and initialized, e.g. a minimal install with a few lines of code or does it require more complex integration and configuration?** -Please refer to https://holmesgpt.dev/installation/cli-installation/ - -**How does an adopter test and validate the installation?** -Please refer to https://holmesgpt.dev/walkthrough/ - -### Security - -**Please provide a link to the project’s cloud native [security self assessment](https://tag-security.cncf.io/community/assessments/).** - -**Please review the [Cloud Native Security Tenets](https://github.com/cncf/tag-security/blob/main/community/resources/security-whitepaper/secure-defaults-cloud-native-8.md) from TAG Security. How are you satisfying the tenets of cloud native security projects?** -This is extremely relevant for us, given the risk that AI models can hallucinate and thereby that HolmesGPT could run malicious commands. To mitigate this, default access is read-only and non-mutating and limited to a pre-approved list of safe commands and integrations. - -**Describe how each of the cloud native principles apply to your project.** -* Make security a design requirement - see above. -* Applying secure configuration has the best user experience - also covered above -* Selecting insecure configuration is a conscious decision - Users must make a conscious and concerted effort to add insecure toolsets (data sources) to HolmesGPT - it cannot be done accidentally. -* Transition from insecure to secure state is possible - users are free to reduce the permissions with which Holmes runs at any point in time and Holmes will identify it and adapt -* Secure defaults are inherited - by default Holmes inherits service roles and permissions from its environment -* Exception lists have first class support - users can add their own toolsets to give Holmes access to additional commands -* Secure defaults protect against pervasive vulnerability exploits - in the case of Holmes, this is equivalent to providing security even when used with malicious/hallucinating LLM which is done as described above -* Security limitations of a system are explainable - Holmes reports permission issues when encountered - -**How do you recommend users alter security defaults in order to "loosen" the security of the project? Please link to any documentation the project has written concerning these use cases.** -https://holmesgpt.dev/data-sources/permissions/ - -**Security Hygiene** -We discuss security implications of features at the design phase, when working on new features. Where warranted there are dedicated discussions around security related aspects. Code reviews function as the final review, but very few issues reach that stage due to thinking about security earlier in the process. - -**Explain the least minimal privileges required by the project and reasons for additional privileges.** -Read-only access to the data sources that are relevant to the requested RCA. - -**Describe how the project is handling certificate rotation and mitigates any issues with certificates.** -Not relevant. - -**Describe how the project is following and implementing [secure software supply chain best practices](https://project.linuxfoundation.org/hubfs/CNCF\_SSCP\_v1.pdf)** -Link is broken, but we strictly review all changes to CI/CD and anything that impacts building the project and distributing it to end users. diff --git a/docs/operator/development.md b/docs/operator/development.md index 9b69a75835..b7aea7798f 100644 --- a/docs/operator/development.md +++ b/docs/operator/development.md @@ -353,4 +353,3 @@ See the main [CLAUDE.md](../../CLAUDE.md) for full contribution guidelines. ## Next Steps - [Configuration](configuration.md) - Understand operator configuration options -- [Architecture Documentation](../adr/operator-initial-architecture.md) - Detailed design decisions diff --git a/docs/operator/index.md b/docs/operator/index.md index c28d1f2c01..235326a6a1 100644 --- a/docs/operator/index.md +++ b/docs/operator/index.md @@ -158,7 +158,7 @@ The Holmes Operator follows the Kubernetes Job/CronJob pattern: - **Operator**: Watches CRDs and orchestrates check execution - **Holmes API**: Executes the actual health check logic using LLM -For detailed architecture information, see the [architecture documentation](../adr/operator-initial-architecture.md). +The operator uses a distributed architecture: a lightweight kopf-based controller handles CRD orchestration and scheduling, while stateless Holmes API servers execute the actual checks. ## Need Help? diff --git a/mkdocs.yml b/mkdocs.yml index fd7e9bd7fe..1165b42617 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -8,7 +8,6 @@ exclude_docs: | _* snippets/ README.md - cncf-self-assesment.md data-sources/permissions.md nav: @@ -33,7 +32,6 @@ nav: - "Health Checks": operator/health-checks.md - "Scheduled Health Checks": operator/scheduled-health-checks.md - "Configuration": operator/configuration.md - - "Architecture": adr/operator-initial-architecture.md - "Development Guide": operator/development.md - "AI Providers": - "Overview": ai-providers/index.md @@ -132,7 +130,6 @@ nav: - "Slash Commands": reference/slash-commands.md - "Troubleshooting": reference/troubleshooting.md - Community: community.md - - "CNCF Self Assessment": cncf-self-assesment.md theme: name: material From 5b0d9efaf8f22ed2c68f92bbc925a00a5fba0f89 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 6 Mar 2026 09:41:58 +0000 Subject: [PATCH 4/4] Add key design decisions to operator architecture section Preserved the still-relevant design decisions from the deleted ADR (Job/CronJob pattern, distributed architecture, APScheduler choice, HealthCheck resources as audit trail) in the operator overview page. https://claude.ai/code/session_01M6qUqTTmhE9kGYkmmaAU3b Signed-off-by: Claude --- docs/operator/index.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/operator/index.md b/docs/operator/index.md index 235326a6a1..43c401f371 100644 --- a/docs/operator/index.md +++ b/docs/operator/index.md @@ -160,6 +160,13 @@ The Holmes Operator follows the Kubernetes Job/CronJob pattern: The operator uses a distributed architecture: a lightweight kopf-based controller handles CRD orchestration and scheduling, while stateless Holmes API servers execute the actual checks. +### Key Design Decisions + +- **Job/CronJob pattern**: Separate HealthCheck and ScheduledHealthCheck CRDs provide clear semantics — one-time checks give immediate feedback, scheduled checks manage recurrence. This mirrors the familiar Kubernetes Job/CronJob model. +- **Distributed operator + API servers**: The operator is a lightweight controller that delegates check execution to stateless Holmes API servers via HTTP. This allows API servers to scale horizontally and isolates scheduling concerns from execution. +- **APScheduler over CronJobs**: Scheduling uses APScheduler within the operator rather than Kubernetes CronJobs. This is more efficient for frequent checks and avoids pod startup overhead per execution. +- **HealthCheck resources as history**: ScheduledHealthCheck creates HealthCheck resources for each run, providing a natural audit trail queryable with standard kubectl commands. + ## Need Help? - **[Join our Slack](https://cloud-native.slack.com/archives/C0A1SPQM5PZ)** - Get help from the community