preset-validation: fix crashes, wire validator into YAML load, validate interactsh_server

[aconite33]

Follow-up to #3058 addressing the issues found in this fuzz comment, plus an additional FQDN check uncovered while testing.

What this PR does

Crash fixes in validate_preset()

• _classify_loc now guards against short loc tuples. validate_preset({"config": {"modules": ["nuclei"]}}) used to raise IndexError; it now returns a clean error.
• The module_dirs pre-pass now skips non-list shapes. validate_preset({"module_dirs": "/tmp/foo"}) used to iterate characters and surface as PermissionError; it now returns a clean type error.
• The modules-list post-pass now skips non-list shapes. validate_preset({"modules": "nuclei"}) used to emit one correct type error plus six bogus per-character lookups; it now emits just the one type error.

Better suggestions for top-level typos

_format_msg now draws extra_forbidden suggestions from PresetSchema field names when len(loc) == 1. Before, top-level typos consulted the dotted config-path universe and produced things like:

• modlues: → "Did you mean 'module_dirs'?"
• flgas: → "Did you mean 'file_blobs'?"
• targest: → "Did you mean 'aggregate'?"

After:

• modlues: → "Did you mean 'modules'?"
• flgas: → "Did you mean 'flags'?"
• targest: → "Did you mean 'target'?"

YAML preset files now run through the validator

validate_preset() is now called from Preset.from_dict(), which covers from_yaml_file, from_yaml_string, and include: directives. Previously the validator only ran for -c CLI args, so typos in -p preset.yml were silently accepted and bad values crashed later inside bake() (e.g., scope.strict: "yesplease" → TypeError from radixtarget; scope: as a list → AttributeError).

interactsh_server is validated as an FQDN or IP

Found while fuzzing: a domain typed without its TLD (a single-label value with no dot) used to pass preset-load validation and then surface as "Failed to register with an interactsh server" warnings from every interactsh-using module mid-scan. A new validate_fqdn_or_ip helper attached as a field_validator to BBOTConfig.interactsh_server rejects this up front. Accepts None, empty string, any IPv4/IPv6, or a hostname containing at least one dot. Rejects single-label values like localhost.

The helper lives in bbot/core/helpers/validators.py alongside validate_host, validate_port, validate_url, and friends. Module authors who want the same check on their own class Config fields can opt in with one line:

from bbot.core.helpers.validators import validate_fqdn_or_ip
from pydantic import field_validator

class Config(BaseModuleConfig):
    host: str = Field("127.0.0.1", description="...")
    _validate_host = field_validator("host")(validate_fqdn_or_ip)

Other fields surveyed but left alone (each carries some compatibility risk for users with internal endpoints, worth its own discussion): web.http_proxy, output module URLs (splunk, elastic, slack, discord, webhook, rabbitmq), database hosts (postgres, mysql, mongo, neo4j), and the baddns custom_nameservers list typing. Each can opt in to validate_fqdn_or_ip (or a future URL validator) in its own module.

flatten_config skips None (latent bug)

While wiring the FQDN validator, surfaced a separate bug in bbot/scanner/preset/environ.py: flatten_config was doing str(v) on every value, which wrote the literal string "None" into BBOT_* env vars for any null field. Pydantic-settings then merged that back into the config dict during validation, defeating any field validator that expects a real value. Now skipped.

validate_preset_file() no longer leaks FileNotFoundError

Catches FileNotFoundError, OSError, and yaml.YAMLError and returns a single PresetValidationError instead.

webbrute schema bumped to accept lists

extensions was declared str but the bundled web/dirbust-heavy.yml preset passes a YAML list, and the runtime feeds it through chain_lists which handles both. The strict YAML validation surfaced this. Schema is now Union[str, list[str]].

CLI exit code: unchanged from baseline

An earlier version of this PR tried to make bbot exit 1 on validation failure. That broke test_cli_customheaders, which asserts _main() returns None on invalid --custom-headers input, and the failure cascaded across the test-distros matrix and Python 3.14. Reverted in commit 02cdec64e. The CLI keeps the existing convention: validation errors are logged ([ERRR] or [WARN]) and _main returns None. Users still see the error; the exit code stays 0 like every other arg-failure path in BBOT.

Tests

• Ten new cases in test_validate_preset.py covering each fix, including the single-label-hostname FQDN repro.
• Four existing tests in test_presets.py updated. They used fake module names (testpreset1...testpreset5, asdf) as opaque config markers; the stricter validator now rejects those, so the tests use real module names with the universal module_timeout field.
• All 40 tests in test_validate_preset.py + test_presets.py + test_config.py + test_cli.py::test_cli_customheaders pass locally.
• Lint and format clean.

Fuzz battery before/after

Surface	Before	After
YAML preset typos	silently accepted	clean error, useful suggestions
config.modules as list	IndexError	clean error
module_dirs as string	PermissionError on /run/docker	clean type error
modules: "nuclei"	type error + 6 bogus cascade	single type error
Top-level typos	bad suggestions from wrong pool	suggestions from PresetSchema
validate_preset_file("/missing")	FileNotFoundError raised	clean error returned
interactsh_server as a single-label hostname (missing TLD)	runtime "Failed to register" cascade from every module	rejected at preset-load with a clean error

Commits

• b10514e17 — Original crash, typo-suggestion, YAML-load wiring, and webbrute-schema fixes.
• 02cdec64e — Revert the cli.py exit-code change to match the existing test framework convention.
• c77dc5588 — interactsh_server FQDN validation + flatten_config None skip + value_error formatter polish + tests.
• a74e065a9 — Cherry-pick of @liquidsec's d423533f4 from retirejs-test-version-range-brittleness. Unrelated to this PR — retire.js's vulnerability DB drifted upstream (CVE-2020-11022 affected-versions range moved from [1.2.0 ... to [1.12.0 ...), breaking the brittle full-string assertion in test_module_retirejs. Picked the existing fix in to get CI green. If that branch lands on preset-validation first, this commit becomes a no-op merge.
• 2d15ac687 — Mechanical move of validate_fqdn_or_ip from bbot/core/config/models.py to bbot/core/helpers/validators.py so it lives next to BBOT's other public validators. No behavior change.

Known item left alone

Pydantic v2 lax bool coercion still accepts "yes" / "true" / "1" / "on" etc. for scope.strict. Flagged in the original review as P1. Not addressed here because adding strict=True to every bool field is a broader change worth its own discussion.

[github-actions]

📊 Performance Benchmark Report

Comparing preset-validation (baseline) vs preset-validation-fixes (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.38ms	4.33ms	-1.1% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.70ms	17.71ms	+0.0% ⚪	✅
Large Closest Match Lookup	360.97ms	357.69ms	-0.9% ⚪	✅
Realistic Closest Match Workload	189.11ms	191.10ms	+1.1% ⚪	✅
Event Memory Medium Scan	1781 B/event	1780 B/event	-0.1% ⚪	✅
Event Memory Large Scan	1768 B/event	1768 B/event	-0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	383.27ms	379.27ms	-1.0% ⚪	✅
Event Validation Full Scan Startup Large Batch	548.99ms	542.13ms	-1.3% ⚪	✅
Make Event Autodetection Small	31.61ms	31.51ms	-0.3% ⚪	✅
Make Event Autodetection Large	322.64ms	320.49ms	-0.7% ⚪	✅
Make Event Explicit Types	14.19ms	14.12ms	-0.5% ⚪	✅
Excavate Single Thread Small	3.818s	3.904s	+2.2% ⚪	✅
Excavate Single Thread Large	9.351s	9.577s	+2.4% ⚪	✅
Excavate Parallel Tasks Small	3.959s	4.016s	+1.4% ⚪	✅
Excavate Parallel Tasks Large	6.416s	6.459s	+0.7% ⚪	✅
Is Ip Performance	3.24ms	3.24ms	+0.2% ⚪	✅
Make Ip Type Performance	11.64ms	11.80ms	+1.4% ⚪	✅
Mixed Ip Operations	4.57ms	4.57ms	-0.1% ⚪	✅
Memory Use Web Crawl	632.7 MB	677.3 MB	+7.0% ⚪	✅
Memory Use Subdomain Enum	35.0 MB	35.0 MB	+0.0% ⚪	✅
Memory Use Deep Chain	8.9 MB	8.9 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	23.1 MB	23.7 MB	+2.7% ⚪	✅
Scan Throughput 100	3.623s	3.591s	-0.9% ⚪	✅
Scan Throughput 1000	28.819s	28.919s	+0.3% ⚪	✅
Typical Queue Shuffle	63.56µs	66.39µs	+4.5% ⚪	✅
Priority Queue Shuffle	723.34µs	742.49µs	+2.6% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 92.68293% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (7bc8c44) to head (2d15ac6).
⚠️ Report is 8 commits behind head on preset-validation.

Files with missing lines	Patch %	Lines
bbot/scanner/preset/validate.py	77%	9 Missing ⚠️

Additional details and impacted files

@@                Coverage Diff                @@
##           preset-validation   #3105   +/-   ##
=================================================
+ Coverage                 90%     90%   +1%     
=================================================
  Files                    452     452           
  Lines                  39341   39437   +96     
=================================================
+ Hits                   35235   35326   +91     
- Misses                  4106    4111    +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.