Preset validation

[TheTechromancer]

Preset validation rework

Replaces silent acceptance of typos and wrong types in BBOT presets with strict, schema-driven validation. Catches mistakes like modlues: [...], scope: {strct: true}, modules: {nucleii: {tgas: "x"}}, and nuclei.mode: aggressive at config-load time instead of after a multi-hour scan produces nothing.

New public API

Validate any preset dict (e.g. from yaml.safe_load) without instantiating a Scanner:

from bbot.scanner import validate_preset, validate_preset_file

errors = validate_preset({
    "modules": ["nuclei"],
    "config": {
        "modules": {
            "nuclei": {"mode": "manual", "ratelimit": 100},
        },
    },
})
if errors:
    for e in errors:
        print(e)   # e.g. [module:nuclei:mode] Expected one of 'manual', ...
    raise SystemExit(1)

# Convenience wrapper for files
errors = validate_preset_file("/path/to/preset.yml")

Returns a list of PresetValidationError objects. Empty list = valid. All errors across all layers are aggregated in a single pass, so a user with five typos sees five errors at once.

The pydantic schemas themselves are also exported, for callers that want type-checking, doc generation, or to validate just one layer (e.g. the global config) on its own:

from bbot.scanner import (
    BBOTConfig,         # global config block (scope/dns/web/engine/deps/…)
    BaseModuleConfig,   # base for every module's `class Config(BaseModuleConfig)`
    PresetSchema,       # top-level preset shape (target, modules, flags, config, …)
    ScopeConfig, DnsConfig, WebConfig, EngineConfig, DepsConfig, DepsToolConfig,
)

These are validation schemas only — they have no defaults of their own (defaults live in bbot/defaults.yml). For full preset validation that also covers per-module Config blocks, use validate_preset(); the composite schema is built dynamically from the loaded module set, so it isn't a static class.

Sample errors

[preset:modlues]                            Unknown option: 'modlues' (value: ['nuclei'])
[config:scope.strct]                        Could not find config option "scope.strct". Did you mean "scope.strict"?
[config:web.http_timeout]                   Expected an integer, got str: 'not-a-number'
[preset:config.modules.nucleii]             Could not find module "nucleii". Did you mean "nuclei"?
[module:nuclei:tgas]                        Unknown option: 'tgas' (value: 'apache')
[module:nuclei:mode]                        Expected one of 'manual', 'technology', 'severe' or 'budget', got 'aggressive'
[module:baddns:custom_nameservers.0]        Expected a string, got int: 1
[config:deps.behavior]                      Expected one of 'abort_on_failure', 'retry_failed', 'ignore_failed', 'disable' or 'force_install', got 'panic'

What changed

Dependency: omegaconf → pydantic + pydantic-settings

omegaconf is gone. Configs are now plain dicts merged with a small deep_update helper, and validation is done by pydantic. pydantic-settings is the new dep; pyyaml is now an explicit (was transitive) dep.

Module schema: class Config(BaseModuleConfig)

Every module's options = {...} + options_desc = {...} pair has been migrated to a typed pydantic class:

# before
options = {"threads": 50, "version": "1.2.5"}
options_desc = {"threads": "How many threads", "version": "httpx version"}

# after
class Config(BaseModuleConfig):
    threads: int = Field(50, description="How many threads")
    version: str = Field("1.2.5", description="httpx version")

BaseModuleConfig carries the three universal options (batch_size, module_threads, module_timeout), so every module accepts those without redeclaring them. 114 modules migrated via codemod (bbot/scripts/migrate_options_to_config.py); a handful tightened by hand to use proper types where they matter:

• nuclei.mode: Literal["manual", "technology", "severe", "budget"]
• baddns.min_severity: Literal["INFO", "LOW", "MEDIUM", "HIGH", "CRITICAL"]
• baddns.min_confidence: Literal["UNKNOWN", "LOW", "MEDIUM", "HIGH", "CONFIRMED"]
• baddns.custom_nameservers: list[str]
• deps.behavior: Literal["abort_on_failure", "retry_failed", "ignore_failed", "disable", "force_install"]

Composite schema, single-pass validation

ModuleLoader.validation_schema builds a composite pydantic model on demand:

FullPresetSchema
  ├─ (PresetSchema fields: target, modules, flags, …)
  └─ config: FullBBOTConfig
              ├─ (BBOTConfig sections: scope, dns, web, engine, deps, …)
              └─ modules: ModulesSchema
                          ├─ nuclei:  NucleiModuleConfig
                          ├─ httpx:   HttpxModuleConfig
                          └─ … one field per loaded module

A single model_validate() call catches typos at every layer. The schema rebuilds when new module dirs are discovered (chicken-and-egg with module_dirs is auto-resolved — validate_preset preloads any custom dirs declared in the preset before validating).

Module Config classes captured via AST + exec

Preload still doesn't import modules (so bbot -l works on hosts missing module deps). The class Config block is captured via ast.get_source_segment, then exec'd at schema-build time in a controlled namespace (Field, BaseModuleConfig, typing.*). Pydantic does the rest. No hand-rolled type whitelist, no annotation-string parsing — anything pydantic understands works (Literal, Union, list[str], etc.).

Cleanup

• BBOTArgs.exclude_from_validation regex and universal_module_options dict deleted; BaseModuleConfig covers their job structurally.
• BBOTArgs.validate() body shrank from ~12 lines of dotted-path lookup to a 4-line validate_preset(...) delegation. CLI typos surface through the same code path that handles preset YAML.
• bbot/scripts/docs.py reads universal-option descriptions from BaseModuleConfig.model_fields directly — no separate constant to keep in sync.
• All omegaconf-specific test helpers and assertions (OmegaConf.merge, omegaconf.errors.ReadonlyConfigError, dot-attribute access) replaced.

Breaking changes

• nuclei.mode rejects unknown values at validation time, not at scan startup with a warning.
• Unknown top-level preset keys (modlues, flgas, etc.) raise ValidationError instead of being silently ignored.
• Unknown module names in modules: / output_modules: / exclude_modules: raise with a closest-match suggestion.
• self.options is no longer populated for migrated modules. Modules that read user-supplied values must use self.config.get(...). (This caught two pre-existing bugs in bbot/modules/templates/gitlab.py and bbot/modules/lightfuzz/lightfuzz.py where module behavior was relying on self.options instead of self.config — both fixed in this PR.)
• The inline ${env:FOO} resolver inside YAML values is gone. Pydantic-settings' native env handling (BBOT_* prefix) is available for whole-field overrides.

Files of interest

• bbot/core/config/models.py — BBOTConfig, PresetSchema, BaseModuleConfig, sub-models. Schema only — defaults live in defaults.yml.
• bbot/core/config/merge.py — deep_update, dotted_get/dotted_set. Replaces OmegaConf.merge/select/from_cli.
• bbot/core/modules.py — _extract_pydantic_config, _exec_config_class, _build_validation_schema, ModuleLoader.validation_schema.
• bbot/scanner/preset/validate.py — validate_preset, validate_preset_file, PresetValidationError. Single-pass aggregator with closest-match suggestions.
• bbot/scripts/migrate_options_to_config.py — codemod that performed the module migration. Idempotent; included for reproducibility.

Test plan

• All test_step_1 config/preset/cli/validate tests pass (39/39 in the touched set).
• Sample module integration tests pass (gitlab_onprem, virustotal, sslcert, robots, crt, httpx, etc.).
• bbot --help, bbot -l, bbot -lp, bbot --current-preset all work and produce the same shape of output as before.
• validate_preset({…}) round-trips: known-good presets return []; presets with seeded typos produce specific, labeled errors.
• Custom module_dirs declared in a preset get preloaded before per-module validation, so user modules aren't falsely flagged.

[github-actions]

🚀 Performance Benchmark Report

⚠️ No current benchmark data available

This might be because:

• Benchmarks failed to run
• No benchmark tests found
• Dependencies missing

[codecov]

Codecov Report

❌ Patch coverage is 95.33195% with 90 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (be1d3b9) to head (6e06458).
⚠️ Report is 17 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/scanner/preset/validate.py	84%	25 Missing ⚠️
bbot/core/modules.py	90%	21 Missing ⚠️
bbot/core/config/models.py	96%	10 Missing ⚠️
bbot/scanner/scanner.py	34%	8 Missing ⚠️
bbot/scanner/preset/args.py	86%	5 Missing ⚠️
bbot/core/config/files.py	74%	4 Missing ⚠️
bbot/core/core.py	90%	3 Missing ⚠️
bbot/core/helpers/misc.py	0%	3 Missing ⚠️
bbot/scanner/preset/preset.py	90%	3 Missing ⚠️
bbot/modules/baddns.py	82%	2 Missing ⚠️
... and 5 more

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3058   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        447     447           
  Lines      40402   40462   +60     
=====================================
+ Hits       36256   36327   +71     
+ Misses      4146    4135   -11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheTechromancer]

Let's consider replacing the auth_required meta boolean with a Pydantic mandatory annotation on individual module config fields. This would identify any modules that can't run without input from the user. Similarly, we should replace the no_secrets_config system with a sensitive Pydantic annotation, to indicate which fields should be hidden/encrypted.

@ausmaster @GabKodes

[aconite33]

Did some fuzzing against this branch with a working preset of mine plus a pile of mutations (typos, wrong types, malformed YAML, hostile inputs to the Python API). The good stuff first: -c key=value, -m / -em / -om / -f / -rf / -ef, and the happy paths through validate_preset() all behave well. Closest-match suggestions are accurate and multi-error aggregation is great. Caught everything I threw at it on those surfaces.

A few things that I think are worth a look before this merges:

YAML preset files bypass the validator entirely

This is the big one. validate_preset() is only wired into args.validate() for -c CLI options. Preset.from_yaml_file → from_dict never invokes it, so the following typos in a -p mypreset.yml are silently accepted:

• top-level: modlues:, flgas:
• config: scope.strct, web.http_timoeut
• unknown module in config.modules.*
• unknown module option (modules.nuclei.tgas)
• wrong type on a module option (modules.nuclei.ratelimit: "abc")

The schema exists and works, but the most common entry point doesn't use it. With this, bad values flow through to bake() and come out as raw tracebacks. For example, scope.strict: "yesplease" in a YAML file crashes with TypeError: argument 'strict_scope': 'str' object cannot be cast as 'bool' from radixtarget; scope: as a list crashes with AttributeError: 'list' object has no attribute 'get'. Calling validate_preset() inside from_dict() (or from_yaml_file()) would catch all of these up front.

validate_preset() crashes on two shape mismatches

Both are reachable from the Python API and would also be reachable from YAML once the validator is wired in:

1. validate_preset({"config": {"modules": ["nuclei"]}}) → IndexError: list index out of range at validate.py:66. pydantic emits loc=('config','modules') (length 2) but _classify_loc indexes parts[2] unconditionally. Needs a length guard.
2. validate_preset({"module_dirs": "/tmp/foo"}) → PermissionError: '/run/docker'. The pre-pass at validate.py:172-178 does for d in source or []. When source is a string it iterates characters and calls module_loader.add_module_dir(d) on each. Should isinstance(source, list) check before iterating.

Character-cascade when a list field gets a string

validate_preset({"modules": "nuclei"}) produces the correct "Expected a list, got str" error and then six bogus per-character lookups:

[preset:modules] Could not find module "n". Did you mean "asn"?
[preset:modules] Could not find module "u". Did you mean "hunt"?
...

The fallback loop at validate.py:197-201 iterates preset_dict[key] without checking that pydantic accepted the shape. Skip the loop when the field's already been flagged as the wrong type.

Suggestions for top-level key typos point at the wrong universe

For extra_forbidden at the preset root, suggestions are drawn from known_paths (the dotted-config universe) rather than PresetSchema field names:

• modlues: → "Did you mean 'module_dirs'?" (should be modules)
• flgas: → "Did you mean 'file_blobs'?" (should be flags)
• targest: → "Did you mean 'aggregate'?" (should be targets)
• output_moduels: → "Did you mean 'module_dirs'?" (should be output_modules)

When len(loc) == 1 and the path doesn't start with config, suggest from PresetSchema.model_fields instead.

Booleans are coerced from strings

config.scope.strict: "yes" / "no" / "true" / "false" / "1" / "0" / "on" / "off" all validate clean (pydantic v2 lax bool). The existing test asserts "yesplease" is rejected, which made me think strict typing was the intent. If so, Field(..., strict=True) on the bool fields would close it. If lax coercion is intentional, ignore.

CLI exits 0 on validation failure

bbot -c scope.strct=true logs [ERRR] and exits 0. cli.py:322 catches BBOTError with log.error() only. CI/scripts driving bbot can't detect validation failures. Probably wants a sys.exit(1) for the validation-error path.

validate_preset_file() raises on a missing file

The return type says list[PresetValidationError] but a missing path leaks FileNotFoundError. Either catch it and return a single error, or document the raise.

[ausmaster]

@TheTechromancer thank you mate