dedup dns children by (rdtype, child) not parent host

[liquidsec]

Summary

DNSResolve.emit_dns_children deduped its outgoing DNS_NAME events with the key (parent_host, rdtype, child_host). Including the parent host in the key meant the same out-of-scope NS/SOA/MX/CNAME value was re-emitted once per in-scope parent that referenced it.

In a recent scan of ~100 corporate domains, this produced:

rdtype	emitted	unique	duplicates
NS	12,072	592	11,480
SOA	3,524	203	3,321
MX	871	123	748
CNAME	854	451	403
TXT	331	36	295
SRV	96	34	62
PTR	96	90	6
TOTAL	17,844	1,529	16,315

beth.ns.cloudflare.com alone was emitted 1,518 times. With Cloudflare and MarkMonitor concentrating many zones onto the same NS pair, the multiplier on consolidated providers is severe.

The 16K duplicate distance-1 events all flowed into every scan module that watches DNS_NAME (dnsbrute, dnscommonsrv, wayback, hunterio, sslcert, excavate, …). Each module had to pull them off its incoming queue, hash them for dedup, run scope and filter_event, and then drop them — serializing queue throughput and slowing scans.

Fix

Drop event.host from the dedup hash. The same (rdtype, child) pair across different parents is the same child event semantically.

- child_hash = hash(f"{event.host}:{module}:{child_host}")
+ child_hash = hash(f"{module}:{child_host}")

Adds TestDNSResolveSharedNameserverDedup to catch regressions: three in-scope domains share an NS/SOA pair; the test asserts each shared nameserver hostname is emitted exactly once across all parents.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs dns-children-dedup-key (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.31ms	4.32ms	+0.4% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	18.51ms	17.52ms	-5.3% ⚪	✅
Large Closest Match Lookup	357.76ms	358.13ms	+0.1% ⚪	✅
Realistic Closest Match Workload	189.03ms	193.33ms	+2.3% ⚪	✅
Event Memory Medium Scan	1394 B/event	1394 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1519 B/event	1519 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	414.89ms	404.22ms	-2.6% ⚪	✅
Event Validation Full Scan Startup Large Batch	547.82ms	531.43ms	-3.0% ⚪	✅
Make Event Autodetection Small	24.75ms	24.96ms	+0.8% ⚪	✅
Make Event Autodetection Large	255.39ms	255.28ms	-0.0% ⚪	✅
Make Event Explicit Types	11.12ms	11.06ms	-0.5% ⚪	✅
Excavate Single Thread Small	3.651s	3.622s	-0.8% ⚪	✅
Excavate Single Thread Large	9.323s	9.385s	+0.7% ⚪	✅
Excavate Parallel Tasks Small	3.724s	3.767s	+1.2% ⚪	✅
Excavate Parallel Tasks Large	6.235s	6.386s	+2.4% ⚪	✅
Intercept Throughput Small	1.059s	1.059s	+0.0% ⚪	✅
Intercept Throughput Medium	1.014s	1.043s	+2.8% ⚪	✅
Is Ip Performance	2.29ms	2.31ms	+0.8% ⚪	✅
Make Ip Type Performance	219.00µs	218.59µs	-0.2% ⚪	✅
Mixed Ip Operations	2.37ms	2.39ms	+1.2% ⚪	✅
Memory Use Web Crawl	398.6 MB	379.8 MB	-4.7% ⚪	✅
Memory Use Subdomain Enum	29.2 MB	29.2 MB	+0.0% ⚪	✅
Memory Use Deep Chain	8.5 MB	8.5 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	20.6 MB	22.6 MB	+10.0% 🟡🟡	⚠️
Scan Throughput 100	3.317s	3.265s	-1.6% ⚪	✅
Scan Throughput 1000	22.769s	23.199s	+1.9% ⚪	✅
Typical Queue Shuffle	5.36µs	5.47µs	+2.0% ⚪	✅
Priority Queue Shuffle	25.97µs	26.29µs	+1.2% ⚪	✅

🎯 Performance Summary

! 1 regression ⚠️
  27 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Parallel Chains: 10.0% 🐌 more memory

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (4bfd96f) to head (354aab1).
⚠️ Report is 27 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3126   +/-   ##
=====================================
- Coverage     90%     90%   -0%     
=====================================
  Files        441     441           
  Lines      38841   38920   +79     
=====================================
+ Hits       34762   34827   +65     
- Misses      4079    4093   +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ausmaster]

Approved under predication that #3141 will be merged along side this as well.