Skip to content

paramminer: dedup by endpoint + param keys, not full URL string - #3180

Merged
liquidsec merged 2 commits into
webbrute-reworkfrom
paramminer-dedup-by-param-keys
Jun 15, 2026
Merged

paramminer: dedup by endpoint + param keys, not full URL string#3180
liquidsec merged 2 commits into
webbrute-reworkfrom
paramminer-dedup-by-param-keys

Conversation

@liquidsec

Copy link
Copy Markdown
Collaborator

Summary

paramminer's input dedup keys on the full URL string. When lightfuzz (or other fuzzers) cycle existing parameters through many probe values, each value mutation creates a URL string paramminer treats as a fresh test surface -- the same endpoint gets wordlist-tested dozens of times for the same effective param structure.

Overrides _incoming_dedup_hash to canonicalize on (event_type, scheme, netloc, path, sorted_param_keys) instead of the full URL. Value mutations of the same endpoint collapse into one test while structural variation (added keys, distinct paths, distinct hosts) remains distinct. paramminer_cookies and paramminer_getparams inherit the fix via class inheritance.

Empirical impact on lightfuzz-heavy presets: ~50% reduction in paramminer test cost, no change to discovered parameters or downstream findings.

@liquidsec
liquidsec force-pushed the paramminer-dedup-by-param-keys branch from 2c5f46b to 6f96b71 Compare June 14, 2026 17:10
@liquidsec
liquidsec changed the base branch from dev to webbrute-rework June 14, 2026 17:10
@github-actions

github-actions Bot commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

🚀 Performance Benchmark Report

⚠️ No current benchmark data available

This might be because:

  • Benchmarks failed to run
  • No benchmark tests found
  • Dependencies missing

@liquidsec
liquidsec force-pushed the paramminer-dedup-by-param-keys branch from 6f96b71 to 27ef0cf Compare June 14, 2026 17:45
@codecov

codecov Bot commented Jun 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.22222% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (b0ac291) to head (b94f583).
⚠️ Report is 8 commits behind head on webbrute-rework.

Files with missing lines Patch % Lines
bbot/modules/paramminer_headers.py 91% 1 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           webbrute-rework   #3180   +/-   ##
===============================================
+ Coverage               90%     90%   +1%     
===============================================
  Files                  449     449           
  Lines                44203   44243   +40     
===============================================
+ Hits                 39516   39569   +53     
+ Misses                4687    4674   -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ausmaster ausmaster added this to the BBOT 3.0 - blazed_elijah milestone Jun 15, 2026
@liquidsec liquidsec mentioned this pull request Jun 15, 2026
28 tasks
@ausmaster
ausmaster self-requested a review June 15, 2026 23:01
@liquidsec
liquidsec force-pushed the paramminer-dedup-by-param-keys branch from b0ac291 to b94f583 Compare June 15, 2026 23:04
@liquidsec
liquidsec merged commit a7a9ef7 into webbrute-rework Jun 15, 2026
12 of 14 checks passed
@liquidsec
liquidsec deleted the paramminer-dedup-by-param-keys branch June 15, 2026 23:05
@liquidsec liquidsec mentioned this pull request Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants