Skip to content

bypass ingress throttle in drain mode - #3134

Merged
liquidsec merged 1 commit into
devfrom
ingress-throttle-drain-mode-bypass
May 30, 2026
Merged

bypass ingress throttle in drain mode#3134
liquidsec merged 1 commit into
devfrom
ingress-throttle-drain-mode-bypass

Conversation

@liquidsec

@liquidsec liquidsec commented May 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

The memory-pressure ingress throttle added in 694ea643c (PR #3083) has a feedback-loop failure mode: when system memory is held high by buffered event objects in queues, the throttle slows the only thing that reduces queue size (ScanIngress), which keeps memory high, which keeps the throttle engaged. In the pathological case (modules killed, only dedup-drop work left to clear), scans get trapped draining at <1 event/sec for hours.

The mechanism

  1. Modules emit a large fleet of heavy events; system memory crosses 90%.
  2. _ingress_delay becomes ~1.9s/event at 91.9% per the linear formula in _compute_ingress_delay.
  3. ScanIngress now sleeps 1.9s before each q.get_nowait() — throughput collapses to ~0.5 events/sec.
  4. Pulled events are mostly dedup-drops (zero memory freed per pull when modules are stopped).
  5. Memory stays at ~92%; next status tick recomputes the same ~1.9s. Goto 3.

User observation: closing an unrelated program (dropping system memory below threshold) caused the scan to immediately zip through hundreds of thousands of queued events. Confirms the throttle was the entire bottleneck.

Fix

In the status-loop recompute (scanner.py), force _ingress_delay = 0.0 when no non-intercept module has work in flight or queued. In that state ingress IS the drain, so throttling it is counterproductive. Predicate checks running OR outgoing_queue OR num_incoming_events to avoid false-clearing during a momentary lull in healthy operation.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Performance Benchmark Report

⚠️ No current benchmark data available

This might be because:

  • Benchmarks failed to run
  • No benchmark tests found
  • Dependencies missing

@codecov

codecov Bot commented May 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (133fbbb) to head (56d517d).
⚠️ Report is 17 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3134   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        441     441           
  Lines      38743   38773   +30     
=====================================
+ Hits       34663   34693   +30     
  Misses      4080    4080           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ausmaster ausmaster left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good.

@liquidsec
liquidsec merged commit 6e3041d into dev May 30, 2026
19 of 20 checks passed
@liquidsec
liquidsec deleted the ingress-throttle-drain-mode-bypass branch May 30, 2026 16:07
@ausmaster ausmaster added this to the BBOT 3.0 - blazed_elijah milestone Jun 1, 2026
@liquidsec liquidsec mentioned this pull request Jun 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working high-priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants