From ac3f9a190848666b6d47efceb37d24da7f209eed Mon Sep 17 00:00:00 2001 From: Christopher Lusk Date: Wed, 1 Apr 2026 15:32:40 -0400 Subject: [PATCH] Add DOS / Resource Exhaustion as Threat 6 to security threat model Addresses #121 with the adjustments proposed in the issue discussion: - Ranked DOS #3 in the threat priority list (between insider threat and agent drift) - Placed the new section after temporal split-payload and before cross-cutting principles - Cross-referenced Threat 2 rate limiting instead of duplicating it - Trimmed defense considerations to focus on agentic-unique defenses - Consolidated overlapping open questions Co-Authored-By: Claude Opus 4.6 --- docs/problems/security-threat-model.md | 71 +++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 2 deletions(-) diff --git a/docs/problems/security-threat-model.md b/docs/problems/security-threat-model.md index 6036b97399..43bdd7b932 100644 --- a/docs/problems/security-threat-model.md +++ b/docs/problems/security-threat-model.md @@ -6,8 +6,9 @@ Defending the agentic system against adversarial attacks. Security is not a feat 1. **External prompt injection** — most immediate, most novel 2. **Insider threat / compromised credentials** — amplified by agent authority -3. **Agent drift** — insidious, slow, hard to detect -4. **Supply chain attacks** — partially addressed by existing tooling, but agentic development introduces a novel trust boundary +3. **Denial of Service / Resource Exhaustion** — cost asymmetry and automated-response amplification make it high-impact +4. **Agent drift** — insidious, slow, hard to detect +5. **Supply chain attacks** — partially addressed by existing tooling, but agentic development introduces a novel trust boundary ## Threat 1: External prompt injection @@ -292,6 +293,72 @@ This means the attack surface includes: - How does this interact with the agent drift problem? Gradual, non-malicious test quality degradation creates the same blind spots that a deliberate attacker would exploit. - What heuristics should agents use to identify opaque/binary files? File extension? Entropy analysis? MIME type detection? How do we avoid blocking legitimate binary test fixtures (e.g., golden files for image processing)? +## Threat 6: Denial of Service (DOS) / Resource Exhaustion + +### The attack + +An attacker triggers excessive consumption of compute, API tokens, or event-processing capacity to degrade or disable the agentic system. Unlike traditional web application DOS which targets request handling, agentic DOS targets the uniquely expensive operations that agents perform — LLM inference, sandbox provisioning, code generation, and multi-agent coordination. + +### Attack vectors + +**Event flooding:** +- Rapidly filing issues, posting comments, toggling labels, or creating PRs to trigger agent invocations at scale +- Abusing slash commands (`/triage`, `/implement`) to queue expensive operations +- Creating issues in bulk across multiple repos in an organization to saturate shared infrastructure + +**Cost amplification:** +- Crafting issues that cause maximum LLM token consumption (extremely long descriptions, requests for exhaustive analysis) +- Triggering implementation agents on problems designed to maximize iteration loops (ambiguous requirements that never converge) +- Exploiting the implement-review feedback loop to cause unbounded cycles between agents +- Filing issues that reference enormous external documents (via URLs) that the agent will attempt to fetch and process + +**Cascade amplification:** +- Injecting events that cause agent-to-agent reaction chains (implementation triggers review, review rejection triggers re-implementation, ad infinitum) +- Exploiting label state machine transitions to create oscillating states that repeatedly trigger agent runs +- Causing concurrent agent invocations that compete for the same resources (file locks, API rate limits) leading to repeated failures and retries + +**Sandbox resource exhaustion:** +- Crafting issues where the described fix requires computationally expensive operations (large-scale refactoring, combinatorial test generation) +- Triggering agents that clone and process very large repositories +- Causing agents to produce extremely large diffs or output artifacts that consume storage + +### Why it's dangerous + +Agentic systems are uniquely vulnerable to DOS because: + +1. **Automated response guarantees amplification.** A human developer ignores spam. An agent configured to respond to qualifying events cannot — every event that matches the trigger criteria produces an expensive response. +2. **Cost is asymmetric.** Filing a GitHub issue is free. Processing it with an LLM agent costs real money (API tokens, compute time). The attacker's cost-to-damage ratio is extremely favorable. +3. **Shared infrastructure multiplies impact.** If the organization uses shared runners or a centralized fullsend repo, DOS against one repository can block agent operations across the entire organization. +4. **Recovery is not instant.** Unlike a web server that recovers when the flood stops, agent backlogs persist — queued work items, partial implementations, and cascading retries may continue consuming resources after the attack ceases. + +### Defense considerations + +Agentic DOS requires defenses beyond standard infrastructure hardening (sandbox resource quotas via cgroups/Kata Containers, compute limits). The following focus on what is unique to this context: + +- **Cost budgets** — set per-repo and per-org budgets for LLM API token consumption. When a budget threshold is reached, require human approval before further agent invocations. +- **Loop circuit breakers** — enforce hard limits on implement-review cycles. The entry point script should enforce these limits deterministically, not rely on the agent's self-restraint. +- **Event debouncing and deduplication** — collapse rapid-fire events on the same issue/PR into a single agent invocation rather than spawning one per event. +- **Tiered response based on actor trust** — events from non-org-members or new contributors could be subject to stricter rate limits or require human approval before triggering agents. +- **Input size limits** — cap the size of issue descriptions, comments, and referenced content that agents will process. Truncate or reject inputs above a threshold. +- **Backpressure mechanisms** — when agent queue depth exceeds a threshold, new events should be rejected or deferred rather than queued, with notification to org administrators. +- **Rate limiting per actor and per repository** — cap the number of agent-triggering events a single user can generate within a time window, and limit concurrent agent runs per repository and per organization. Note: [Threat 2](#threat-2-insider-threat--compromised-credentials) discusses rate limiting for anomaly detection of compromised credentials via behavioral patterns. DOS rate limiting is distinct — it caps event volume from any actor regardless of intent, as a resource protection mechanism rather than a compromise detection signal. + +### Intersection with other threats + +DOS has elements that touch several existing threats: +- **Prompt injection** — a prompt injection payload could instruct the agent to perform expensive operations, turning a prompt injection into a DOS amplification +- **Agent-to-agent injection** — a compromised agent could generate outputs designed to trigger expensive cascades in downstream agents +- **Insider threat** — a compromised account with org membership bypasses actor-based rate limits +- **Agent drift** — gradual increases in agent response time or resource consumption may indicate an unintentional DOS caused by system degradation + +### Open questions + +- Should cost budgets trigger a hard stop or a human-in-the-loop approval flow? +- How do we distinguish legitimate bursts of activity (e.g., a major outage generating many related bug reports) from an attack, and should rate limits be configurable per organization to account for this? +- How do we handle the case where rate limiting causes legitimate high-priority issues to be delayed? +- Can we implement cost estimation before committing to an agent run — predicting whether an issue will require expensive processing and routing accordingly? +- Should the event debouncing strategy from the March 31 concurrency discussion be treated as a DOS defense or purely a correctness concern? (It serves both purposes.) + ## Cross-cutting security principles 1. **Defense in depth** — no single control should be the only thing preventing an attack