Skip to content

feat(webhooks): SIEM enrichment fields on MemoryDefenseEventData - #2157

Closed
chrislatimer wants to merge 3 commits into
mainfrom
feat/memory-defense-event-siem-enrichment
Closed

feat(webhooks): SIEM enrichment fields on MemoryDefenseEventData#2157
chrislatimer wants to merge 3 commits into
mainfrom
feat/memory-defense-event-siem-enrichment

Conversation

@chrislatimer

@chrislatimer chrislatimer commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds optional SIEM-enrichment fields to MemoryDefenseEventData AND wires the OSS regex defense to populate them. SIEM receivers now get a fingerprinted preview of every matched secret on every memory_defense.triggered delivery — straight from OSS, no extension required.

Schema changes (commit 1)

Five optional fields on MemoryDefenseEventData so downstream extensions (e.g. hindsight-cloud) can surface the per-decision context SIEM operators need to act on a leaked-secret webhook, without a separate event type:

  • severity: low|medium|high|critical
  • api_key_name: human-readable name of the Hindsight API key that submitted the retain — so InfoSec can identify the agent / integration responsible
  • hits: list of MemoryDefenseHit { detector, preview } per match, where preview is a fingerprinted rendering (ghp_AAAA...BBBB). Never the raw secret; MemoryDefenseHit validates both fields present so an extension can't accidentally post the raw value as the only field
  • memory_unit_id: drill-down pointer for REDACT decisions
  • receipt_uri: pointer into the audit trail

All five fields default to None — existing OSS receivers see no shape change. Receivers should treat absence as "not provided" rather than "no match".

OSS populates hits[] (commit 2)

Three changes to surface fingerprinted previews on the OSS payload itself, so OSS users have SIEM-actionable data without needing the cloud extension:

  • New _fingerprint_value helper with length-aware bucketing (< 6 chars: redact entirely; 6–15: first-2 + ellipsis + last-2; > 15: first-4 + ellipsis + last-4). Raw value never appears in output.
  • apply_redaction returns a hits list — one entry per matched substring (so two GitHub tokens in the same content produce two hits, not a single collapsed label).
  • Orchestrator's _fire_memory_defense_webhook translates the decision's raw hit dicts into MemoryDefenseHit entries on the wire. None when no per-hit data exists so receivers can distinguish "no preview info" from a hypothetical empty list.

Motivation

This closes the gap that previously motivated keeping a separate memory_defense.violation event in cloud (before the consolidation that landed in #2077). Cloud and OSS can now ship the same SIEM-actionable payload through the canonical memory_defense.triggered envelope — one event type, one receiver code path, full enrichment from whichever layer is active.

Test plan

  • Schema tests in test_webhooks.py:
    • Five base fields round-trip; five new fields default to None.
    • Enrichment fields populate cleanly through WebhookEvent JSON serialisation.
    • MemoryDefenseHit validates both detector and preview are present.
  • Fingerprinting unit tests in test_memory_defense.py:
    • _fingerprint_value across all three length buckets (parametrized).
    • apply_redaction produces per-match fingerprinted previews; raw secret never present.
    • Multiple matches of the same pattern produce multiple hits.
  • Extended test_screen_redacts_secret to assert the regex extension passes hits onto DefenseDecision.
  • Extended test_retain_fires_webhook_on_redact to assert the wire payload carries hits[] with the correct fingerprint.
  • Existing memory-defense webhook flow tests still green: test_retain_fires_webhook_on_block.
  • Full test_memory_defense.py + test_webhooks.py: 100 passed.
  • ruff check + ty check clean.

Adds five optional fields to MemoryDefenseEventData so downstream extensions
(e.g. hindsight-cloud) can surface the per-decision context SIEM operators
need to act on a leaked-secret webhook: severity, the API key that submitted
the retain, fingerprinted hit previews for correlation against credential
inventories, and pointers into the audit trail.

Backward-compatible: all five fields default to None and OSS's built-in
regex defense leaves them unset, so existing OSS receivers see no shape
change. Receivers should treat absence as "not provided" rather than "no
match" — the OSS path still populates matched_types as before.

The hit preview is wrapped in a new MemoryDefenseHit model whose docstring
pins the rule that preview must be a fingerprinted rendering of the value,
never the raw secret. Validates that both detector and preview are present
to guard against extensions accidentally posting the raw value as the only
field.

Closes the gap that motivated keeping a separate memory_defense.violation
event in cloud before the recent consolidation: cloud can now ship the
same SIEM-actionable payload through the canonical memory_defense.triggered
envelope.
@chrislatimer
chrislatimer force-pushed the feat/memory-defense-event-siem-enrichment branch from e8216a2 to a74845e Compare June 12, 2026 02:12
Builds on the schema added in the previous commit by populating the
SIEM-relevant hits field from the OSS regex defense. SIEM receivers
now get a per-match preview (e.g. ghp_AAAA...AAAA) for every redaction
the OSS extension fires, in addition to the existing matched_types list.

Three changes:

1. New _fingerprint_value helper produces a length-aware redaction-
   identifiable rendering of a matched value:
   - length < 6:  returns "[redacted]" (avoid leaking material on
     short matches like an isolated -----BEGIN... marker)
   - length 6-15: first-2 + ellipsis + last-2
   - length > 15: first-4 + ellipsis + last-4
   The raw value never appears in the output.

2. apply_redaction returns a hits list alongside matched_types - one
   entry per matched substring (so two GitHub tokens produce two hits
   rather than collapsing into a single label). Hits threaded through
   RedactionResult -> DefenseDecision -> MemoryDefenseEventData via the
   orchestrator's fire helper.

3. _fire_memory_defense_webhook translates the decision's raw hit dicts
   into MemoryDefenseHit entries. None when the decision carries no
   per-hit data so receivers can distinguish "no preview info" from
   a hypothetical empty list.

Test plan:
- New unit tests for _fingerprint_value across all three length
  buckets (parametrized) plus apply_redaction shape: per-match
  fingerprinted previews, raw value never present, multiple matches
  of the same pattern produce multiple hits.
- Extended test_screen_redacts_secret to assert the regex extension
  passes hits onto DefenseDecision.
- Extended test_retain_fires_webhook_on_redact to assert the wire
  payload carries hits[].
- Helper _memory_defense_webhook_events now orders most-recent-first
  so events[0] always reflects the latest delivery.
- Full test_memory_defense.py + test_webhooks.py: 100 passed.
- ruff + ty: clean.
@chrislatimer
chrislatimer deleted the feat/memory-defense-event-siem-enrichment branch June 12, 2026 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant