feat(webhooks): SIEM enrichment fields on MemoryDefenseEventData - #2157
Closed
chrislatimer wants to merge 3 commits into
Closed
feat(webhooks): SIEM enrichment fields on MemoryDefenseEventData#2157chrislatimer wants to merge 3 commits into
chrislatimer wants to merge 3 commits into
Conversation
Adds five optional fields to MemoryDefenseEventData so downstream extensions (e.g. hindsight-cloud) can surface the per-decision context SIEM operators need to act on a leaked-secret webhook: severity, the API key that submitted the retain, fingerprinted hit previews for correlation against credential inventories, and pointers into the audit trail. Backward-compatible: all five fields default to None and OSS's built-in regex defense leaves them unset, so existing OSS receivers see no shape change. Receivers should treat absence as "not provided" rather than "no match" — the OSS path still populates matched_types as before. The hit preview is wrapped in a new MemoryDefenseHit model whose docstring pins the rule that preview must be a fingerprinted rendering of the value, never the raw secret. Validates that both detector and preview are present to guard against extensions accidentally posting the raw value as the only field. Closes the gap that motivated keeping a separate memory_defense.violation event in cloud before the recent consolidation: cloud can now ship the same SIEM-actionable payload through the canonical memory_defense.triggered envelope.
chrislatimer
force-pushed
the
feat/memory-defense-event-siem-enrichment
branch
from
June 12, 2026 02:12
e8216a2 to
a74845e
Compare
Builds on the schema added in the previous commit by populating the
SIEM-relevant hits field from the OSS regex defense. SIEM receivers
now get a per-match preview (e.g. ghp_AAAA...AAAA) for every redaction
the OSS extension fires, in addition to the existing matched_types list.
Three changes:
1. New _fingerprint_value helper produces a length-aware redaction-
identifiable rendering of a matched value:
- length < 6: returns "[redacted]" (avoid leaking material on
short matches like an isolated -----BEGIN... marker)
- length 6-15: first-2 + ellipsis + last-2
- length > 15: first-4 + ellipsis + last-4
The raw value never appears in the output.
2. apply_redaction returns a hits list alongside matched_types - one
entry per matched substring (so two GitHub tokens produce two hits
rather than collapsing into a single label). Hits threaded through
RedactionResult -> DefenseDecision -> MemoryDefenseEventData via the
orchestrator's fire helper.
3. _fire_memory_defense_webhook translates the decision's raw hit dicts
into MemoryDefenseHit entries. None when the decision carries no
per-hit data so receivers can distinguish "no preview info" from
a hypothetical empty list.
Test plan:
- New unit tests for _fingerprint_value across all three length
buckets (parametrized) plus apply_redaction shape: per-match
fingerprinted previews, raw value never present, multiple matches
of the same pattern produce multiple hits.
- Extended test_screen_redacts_secret to assert the regex extension
passes hits onto DefenseDecision.
- Extended test_retain_fires_webhook_on_redact to assert the wire
payload carries hits[].
- Helper _memory_defense_webhook_events now orders most-recent-first
so events[0] always reflects the latest delivery.
- Full test_memory_defense.py + test_webhooks.py: 100 passed.
- ruff + ty: clean.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional SIEM-enrichment fields to
MemoryDefenseEventDataAND wires the OSS regex defense to populate them. SIEM receivers now get a fingerprinted preview of every matched secret on everymemory_defense.triggereddelivery — straight from OSS, no extension required.Schema changes (commit 1)
Five optional fields on
MemoryDefenseEventDataso downstream extensions (e.g. hindsight-cloud) can surface the per-decision context SIEM operators need to act on a leaked-secret webhook, without a separate event type:severity:low|medium|high|criticalapi_key_name: human-readable name of the Hindsight API key that submitted the retain — so InfoSec can identify the agent / integration responsiblehits: list ofMemoryDefenseHit { detector, preview }per match, wherepreviewis a fingerprinted rendering (ghp_AAAA...BBBB). Never the raw secret;MemoryDefenseHitvalidates both fields present so an extension can't accidentally post the raw value as the only fieldmemory_unit_id: drill-down pointer for REDACT decisionsreceipt_uri: pointer into the audit trailAll five fields default to
None— existing OSS receivers see no shape change. Receivers should treat absence as "not provided" rather than "no match".OSS populates
hits[](commit 2)Three changes to surface fingerprinted previews on the OSS payload itself, so OSS users have SIEM-actionable data without needing the cloud extension:
_fingerprint_valuehelper with length-aware bucketing (< 6 chars: redact entirely; 6–15: first-2 + ellipsis + last-2; > 15: first-4 + ellipsis + last-4). Raw value never appears in output.apply_redactionreturns ahitslist — one entry per matched substring (so two GitHub tokens in the same content produce two hits, not a single collapsed label)._fire_memory_defense_webhooktranslates the decision's raw hit dicts intoMemoryDefenseHitentries on the wire.Nonewhen no per-hit data exists so receivers can distinguish "no preview info" from a hypothetical empty list.Motivation
This closes the gap that previously motivated keeping a separate
memory_defense.violationevent in cloud (before the consolidation that landed in #2077). Cloud and OSS can now ship the same SIEM-actionable payload through the canonicalmemory_defense.triggeredenvelope — one event type, one receiver code path, full enrichment from whichever layer is active.Test plan
test_webhooks.py:None.WebhookEventJSON serialisation.MemoryDefenseHitvalidates bothdetectorandprevieware present.test_memory_defense.py:_fingerprint_valueacross all three length buckets (parametrized).apply_redactionproduces per-match fingerprinted previews; raw secret never present.test_screen_redacts_secretto assert the regex extension passes hits ontoDefenseDecision.test_retain_fires_webhook_on_redactto assert the wire payload carrieshits[]with the correct fingerprint.test_retain_fires_webhook_on_block.test_memory_defense.py+test_webhooks.py: 100 passed.ruff check+ty checkclean.