From 2983a9ba1a9f680d033fc35cab1d63a6bc9698dd Mon Sep 17 00:00:00 2001 From: liquidsec Date: Fri, 22 May 2026 16:56:15 -0400 Subject: [PATCH 01/29] add 2.x -> 3.0 migration guide --- README.md | 4 + docs/migration/3.0_breaking_changes.md | 387 +++++++++++++++++++++++++ mkdocs.yml | 2 + 3 files changed, 393 insertions(+) create mode 100644 docs/migration/3.0_breaking_changes.md diff --git a/README.md b/README.md index c04d8f9ae1..b6ba810220 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,8 @@ pipx install --pip-args '\--pre' bbot _For more installation methods, including [Docker](https://hub.docker.com/r/blacklanternsecurity/bbot), see [Getting Started](https://www.blacklanternsecurity.com/bbot/Stable/)_ +> **Upgrading from 2.x?** BBOT 3.0 contains breaking changes to the CLI, presets, modules, events, and Python API. See the [2.x → 3.0 Migration Guide](https://www.blacklanternsecurity.com/bbot/Stable/migration/3.0_breaking_changes/) ([source](docs/migration/3.0_breaking_changes.md)) before upgrading. + > **Speed tip:** BBOT's DNS engine spins up ten workers per resolver in `/etc/resolv.conf`. Adding more unfiltered resolvers dramatically speeds up scans. See the [sample resolv.conf](docs/data/resolv-sample.conf) and [Tips and Tricks](https://www.blacklanternsecurity.com/bbot/Stable/scanning/tips_and_tricks/#speed-up-scans-with-more-dns-resolvers) for details. ## Example Commands @@ -388,6 +390,8 @@ For details, see [Configuration](https://www.blacklanternsecurity.com/bbot/Stabl - [Nuclei](https://www.blacklanternsecurity.com/bbot/Stable/modules/nuclei) - [Custom YARA Rules](https://www.blacklanternsecurity.com/bbot/Stable/modules/custom_yara_rules) - [Lightfuzz (DAST)](https://www.blacklanternsecurity.com/bbot/Stable/modules/lightfuzz) + - **Migration** + - [2.x → 3.0 Breaking Changes](https://www.blacklanternsecurity.com/bbot/Stable/migration/3.0_breaking_changes) - **Misc** - [Contribution](https://www.blacklanternsecurity.com/bbot/Stable/contribution) - [Release History](https://www.blacklanternsecurity.com/bbot/Stable/release_history) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md new file mode 100644 index 0000000000..7ba0a44095 --- /dev/null +++ b/docs/migration/3.0_breaking_changes.md @@ -0,0 +1,387 @@ +# Migrating from BBOT 2.x to 3.0 + +BBOT 3.0 is a major release with a wide-ranging cleanup of the CLI, preset +syntax, event API, and module ecosystem. This page enumerates everything that +changed in a backwards-incompatible way so existing scans, custom modules, and +integrations can be updated. + +If a change is missing here, please open an issue. + +--- + +## CLI / Targeting + +The `--whitelist` concept was retired. BBOT now distinguishes between **target** +(what defines scope, i.e. what `in_target()` checks) and **seeds** (the events +that actually drive passive modules at scan start). + +| 2.x | 3.0 | +|-----|-----| +| `-t / --targets` (seed + scope) | `-t / --targets` (scope only) | +| `-w / --whitelist` | removed (positional `-t` now plays this role) | +| *(no equivalent)* | `-s / --seeds` (optional override of seed events) | +| `-s / --silent` | `-S / --silent` (capitalized; `-s` reassigned to `--seeds`) | +| `--allow-deadly` | removed (see flag changes below) | + +Behavior: + +- If `--seeds` is omitted, seeds default to whatever was passed to `--targets`, + so the simple case (`bbot -t evilcorp.com -p subdomain-enum`) is unchanged. +- `--strict-scope` now means "only this exact host, no subdomains" against the + target, not the whitelist. +- The `Preset(...)` Python API mirrors the CLI: positional args are the + **target**, and `seeds=` is a keyword. `whitelist=` is gone. + +### Python API + +```python +# 2.x +preset = Preset("evilcorp.com", whitelist=["evilcorp.com", "evilcorp.net"]) +scan.whitelist # Target object +scan.whitelisted(host) + +# 3.0 +preset = Preset("evilcorp.com", "evilcorp.net", seeds=["evilcorp.com"]) +scan.target # BBOTTarget wrapper +scan.target.target # the radix-backed scope +scan.target.seeds # seed events +scan.in_target(host) +``` + +`Scanner.whitelist` and `Scanner.whitelisted()` have been removed. +`BBOTTarget.whitelist` is now `BBOTTarget.target`. Pickling of `BBOTTarget` was +also removed. + +--- + +## Flags + +| Removed | Replacement / notes | +|---------------------|---------------------| +| `aggressive` | use `loud` (network volume) and/or `invasive` (destructive) | +| `deadly` | dropped along with `--allow-deadly`; affected modules now live under regular flags | +| `noisy` | renamed to `loud` | +| `web-basic` | renamed to `web` | +| `web-thorough` | renamed to `web-heavy` | + +| Added | Meaning | +|-------------|---------| +| `invasive` | Intrusive or potentially destructive | +| `safe` | Non-intrusive and non-destructive (now enforced on every module) | +| `download` | Modules that download files, apps, or repositories | +| `web-heavy` | More advanced web scanning functionality | + +Every module must now declare at least one of `passive` / `active` **and** at +least one of `safe`, `loud`, or `invasive`. Custom modules carrying the removed +flags will fail validation. + +--- + +## Presets + +### Renamed + +| 2.x | 3.0 | +|----------------------|--------------------| +| `web-basic` | `web` | +| `nuclei-intense` | `nuclei-heavy` | +| `spider-intense` | `spider-heavy` | + +### Removed + +- `web-thorough` (use `web-heavy`) +- `baddns-intense` (the baddns preset tiers were rebuilt; use `baddns` / + `baddns-heavy`) +- `web/lightfuzz-medium`, `web/lightfuzz-superheavy` (lightfuzz preset tiers + were rebuilt; see `web/lightfuzz`, `web/lightfuzz-heavy`, + `web/lightfuzz-max`) + +### Added + +`baddns`, `baddns-heavy`, `web-heavy`, `web/lightfuzz`, `web/lightfuzz-max`, +`web/paramminer-heavy`. + +### Syntax + +Preset YAML files now accept the singular key `target:` in addition to +`targets:` (both are merged). The old `whitelist:` key is gone; use `seeds:` if +you need seeds that differ from the target. Lines beginning with `#` inside +target / seed / blacklist lists are stripped as comments. File paths in those +lists are resolved relative to the preset file via the new `PresetPath` +mechanism. + +The top-level scan options that used to be implicit are now required to live +under `config:` inside a preset (this was previously documented behavior, now +enforced and called out in `defaults.yml`). + +--- + +## Modules + +### Removed (no direct replacement) + +- `azure_realm` +- `digitorus` +- `passivetotal` +- `sitedossier` +- `smuggler` +- `vhost` +- `wappalyzer` + +### Removed and replaced + +| 2.x | 3.0 replacement | +|------------------|-----------------| +| `httpx` | `http` | +| `ffuf` | `webbrute` | +| `ffuf_shortnames`| `webbrute_shortnames` | +| `bucket_azure` | `bucket_microsoft` | +| `extractous` | `kreuzberg` | +| `output.http` | `output.webhook` | +| `censys` (single module) | split into `censys_dns` and `censys_ip` | + +If you used any of these in a custom preset or `-m` / `-em` invocation, update +the module name accordingly. The `--list-modules` output is the source of +truth. + +!!! warning "The `http` name was reassigned" + The module called `http` in 3.0 is **not** the same as the output module + called `http` in 2.x. The name was reassigned: + + - **`http` (3.0, scan module)** — the replacement for the old `httpx` + scan module. Probes URLs over the shared in-process blasthttp client. + - **`output.http` (2.x, output module)** — the webhook-style output module + that POSTed events to an arbitrary HTTP endpoint. This is now + `output.webhook` in 3.0. + + Old scans that ran `bbot -om http ...` were emitting events to a webhook; + in 3.0 the equivalent is `bbot -om webhook ...`. The new `bbot -m http ...` + is a scan module that probes URLs, which is a completely different thing. + +### New modules worth knowing about + +- `http` — replaces `httpx`; runs through the in-process [blasthttp](https://github.com/blacklanternsecurity/blasthttp) client. +- `webbrute` / `webbrute_shortnames` — ffuf replacements, also via blasthttp. +- `bucket_hetzner`, `shodan_enterprise`, `trajan`, `legba`. +- Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq`. +- Lightfuzz submodules `esi` and `ssrf` (the old `generic_ssrf` module was + deleted in favor of the lightfuzz submodule). + +### Module API + +- `BaseModule` no longer has `_event_handler_watchdog_task` as a class + attribute and the watchdog is owned by `_setup()`. +- New `BaseModule.update_event(event, **kwargs)` and `emit_event(existing_event, + ...)` flow. Passing an existing event to `make_event()` now raises — call + `update_event()` instead. +- New `BaseModule.setup_deps()` lifecycle hook (runs alongside `setup()` for + pure dependency installation like AI models or wordlists). +- New `_disable_auto_module_deps = True` opt-out so a module that watches URLs + doesn't automatically pull in `http`/`blasthttp`. +- New `accept_seeds` attribute. Defaults to `True` for passive modules, + `False` otherwise. Override explicitly if you want different behavior. +- `default_discovery_context` now uses `{event.pretty_string}` instead of + `{event.data}` — the latter is now a dict for URL-like events (see below). + +--- + +## Events + +### Removed event types + +- `VULNERABILITY` is gone. Emit a `FINDING` with `severity` set to + `"CRITICAL"`, `"HIGH"`, `"MEDIUM"`, `"LOW"`, or `"INFO"` instead. The pseudo + event type `TARGET` was also retired in favor of `SEED`. + +### Class hierarchy + +| 2.x | 3.0 | +|-----|-----| +| `URL_UNVERIFIED(BaseEvent)` — data is a string | `URL_UNVERIFIED(DictHostEvent)` — data is a dict with `url`, `path`, etc. | +| `URL(URL_UNVERIFIED)` — string data | `URL(URL_UNVERIFIED)` — dict data | +| `STORAGE_BUCKET(DictEvent, URL_UNVERIFIED)` | `STORAGE_BUCKET(URL_UNVERIFIED)` | +| `HTTP_RESPONSE(URL_UNVERIFIED, DictEvent)` | `HTTP_RESPONSE(URL_UNVERIFIED)` | +| `DictPathEvent(DictEvent)` | `DictPathEvent(DictHostEvent)` | +| *(no UDP event)* | `OPEN_UDP_PORT(OPEN_TCP_PORT)` | + +URL events are now dict-backed. Reading `.data` on a `URL` / `URL_UNVERIFIED` / +`HTTP_RESPONSE` returns a dict, not a string. Use the new `event.url` property +to get the URL string, or `event.pretty_string` for a human-readable form. +Many built-in modules and most tests were updated accordingly; custom modules +that compared or formatted `event.data` for URL events must be updated. + +### Removed attributes + +- `event.confidence` and `event.cumulative_confidence` no longer exist on + `BaseEvent`. Confidence is now a per-finding value carried inside + `FINDING.data["confidence"]` and validated against + `("UNKNOWN", "LOW", "MEDIUM", "HIGH", "CONFIRMED")`. +- `source_domain` has been removed from events. +- `_always_emit_tags`: the literal tag `"target"` was renamed to `"seed"`. +- Tags `ip-
`, `http-title-`, and `cloud-<type>` are no longer + emitted as event tags by the `http` / cloudcheck pipelines. Use + `event.resolved_hosts`, `event.host_metadata`, and the simplified cloud tags + (`cloud`, plus a single provider tag) instead. + +### New event surface + +- `event.url` — string URL property (works on URL-like events, returns `""` + otherwise). +- `event.pretty_string` — human-readable representation, used in logs and + discovery context. +- `event.host_metadata` — dict of structured per-host metadata (cloud + providers, ASN info, etc.). Replaces the long-tail of `cloud-*` tags. +- Mutation helpers `add_resolved_host()`, `update_resolved_hosts()`, + `add_dns_child()`, `set_raw_dns_record()`. Direct assignment to the backing + slots (`event._resolved_hosts = ...`, `event.dns_children["A"].add(...)`) is + no longer the public contract: the slots are lazily initialized to `None` + for memory reasons and exposed read-only via properties. + +### FINDING severity / confidence + +`FINDING.data` is now validated against a fixed allowlist: + +- `severity`: `"INFO" | "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"` + (the old `"INFORMATIONAL"` and `"MODERATE"` values have been renamed to + `INFO` and `MEDIUM`) +- `confidence`: `"UNKNOWN" | "LOW" | "MEDIUM" | "HIGH" | "CONFIRMED"` + +Severity and confidence are recorded on the event as `severity-<level>` and +`confidence-<level>` tags. Lowercase tags like `high` or `medium` are no +longer added directly. + +### ASN as a target type + +`ASN:12345` (or `AS12345`) can now be used as a scan target — the seed will be +expanded to its registered CIDRs via the `asndb` library at scan start. ASN +lookups are also wired through the new `bbot_io_api_key` config (or +`BBOT_IO_API_KEY` env var). + +--- + +## Config + +`bbot/defaults.yml` saw a number of breaking renames and additions: + +### Renamed + +| 2.x | 3.0 | +|---------------------------|---------------------------| +| `web.httpx_timeout` | `web.blasthttp_timeout` | +| `web.httpx_retries` | `web.blasthttp_retries` | +| `dns.threads` (global) | `dns.threads` (now per-resolver; default lowered from 25 → 10) | + +### Removed + +- `deps.ffuf.version` (ffuf is gone). +- The top-level `modules.json.siem_friendly` flag (and the `siem_friendly` + output module). + +### Added + +- `max_mem_percent` — global ingress throttle when RSS exceeds the threshold. +- `web.user_agent_suffix` — appended to the user agent (previously buried as a + hidden CLI flag). +- `web.http_rate_limit` — global rps cap across the shared blasthttp client. +- `web.body_spill.{enabled,cache_mb,compress}` — disk-spill HTTP response + bodies to keep them off the Python heap. +- `dns.cache_size` — DNS LRU size. +- `bbot_io_api_key` — API key for bbot.io services (currently used by ASN + lookups). +- `dns.abort_threshold` default lowered from `50` → `10`. +- `dns.filter_ptrs` semantics: PTR-derived hostnames are now treated as + affiliates by default rather than being injected as in-scope DNS_NAME events + during IP-range scans. + +--- + +## DNS and HTTP internals + +Both the DNS and HTTP engines were rewritten and the old subprocess +architecture was deleted. + +- **DNS**: `bbot/core/helpers/dns/engine.py` and `dns/mock.py` were removed. + The `DNSHelper` no longer inherits from `EngineClient`; resolution now goes + through the native [blastdns](https://github.com/blacklanternsecurity/blastdns) + Rust client. The `self.helpers.dns.resolver` `dnspython` resolver is gone. +- **HTTP**: `bbot/core/helpers/web/client.py` and `web/engine.py` were + removed. `WebHelper` no longer inherits from `EngineClient`. All HTTP goes + through the shared `self.helpers.blasthttp` client. The httpx-based + `request_batch` / `request_custom_batch` / `curl` methods were replaced by + `request()`, `request_batch_stream(urls, threads=10, **kwargs)`, and + `download()`. +- The blasthttp dependency line is `blasthttp>=0.7.0`. + +Modules that previously instantiated their own `httpx.AsyncClient` or built +custom curl invocations must switch to `self.helpers.request(...)` / +`self.helpers.blasthttp`. + +--- + +## Output and DB models + +- The pydantic / SQLModel models moved from `bbot.db.sql.models` to + `bbot.models.sql` (plus a new `bbot.models.pydantic` / `bbot.models.helpers` + split). External importers must update. +- `output.http` was renamed to `output.webhook`. +- `output.json` lost the `siem_friendly` option. +- New output modules: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, + `zeromq`. +- Neo4j output now also serializes `host_metadata`. + +--- + +## Dependencies and tooling + +- **Build system**: migrated from Poetry to [uv](https://docs.astral.sh/uv/) + + hatchling. `pyproject.toml` is now PEP-621. `poetry.lock` is replaced by + `uv.lock`. Dev install: `uv sync --group dev`. The + `poetry-dynamic-versioning` integration is gone; version is now plain + `3.0.0`. +- **License**: changed from `GPL-3.0` to `AGPL-3.0`. +- **Python**: minimum bumped from `3.9` to `3.10`. Upper bound is now + `<3.15`. +- **Lockstep deps**: + - `radixtarget >=4.0.1,<5` (composition pattern, no longer subclassed) + - `cloudcheck >=10.0.0,<11` + - `blasthttp >=0.7.0` (new) + - `blastdns >=1.9.0,<2` (new) + - `asndb >=1.0.4` (new) + - `zstandard` (new; used by HTTP body spill) + - `httpx` **removed** as a runtime dep. + +Custom modules that imported `httpx`, `dns.asyncresolver`, `radixtarget` +subclasses, or `bbot.db.sql.models` will need to be ported. + +--- + +## Quick migration checklist + +- [ ] Replace `--whitelist` / `whitelist=` with positional target args; only + reach for `--seeds` / `seeds=` when you genuinely want a seed that isn't + in scope. +- [ ] Replace `-s` with `-S` for silent runs. +- [ ] Drop `--allow-deadly`. +- [ ] Rename module references: `httpx → http`, `ffuf → webbrute`, + `bucket_azure → bucket_microsoft`, `extractous → kreuzberg`, + `output.http → output.webhook`, `censys → censys_dns / censys_ip`. +- [ ] Rename preset references: `web-basic → web`, `web-thorough → web-heavy`, + `*-intense → *-heavy`. +- [ ] Rename module flags: `noisy → loud`, `web-basic → web`, + `web-thorough → web-heavy`. Drop `aggressive` / `deadly`. Add `safe` / + `loud` / `invasive` to satisfy the new validation rule. +- [ ] Replace any `VULNERABILITY` emit with a `FINDING` carrying + `severity=...`. +- [ ] Update FINDING severity strings: `INFORMATIONAL → INFO`, + `MODERATE → MEDIUM`. Add a `confidence` field from the new allowlist. +- [ ] Stop reading `event.data` for URL events; use `event.url` / + `event.pretty_string`. Drop dependencies on `event.confidence` / + `cumulative_confidence`. +- [ ] Stop mutating `event._resolved_hosts` / `event.dns_children` directly; + use the new `add_resolved_host` / `add_dns_child` helpers. +- [ ] Update config keys: `web.httpx_timeout → web.blasthttp_timeout`, + `web.httpx_retries → web.blasthttp_retries`. +- [ ] Update imports: `bbot.db.sql.models → bbot.models.sql`. +- [ ] If you ship a custom module that watches URLs but doesn't want + blasthttp auto-enabled, set `_disable_auto_module_deps = True`. +- [ ] Switch your install/build pipeline from Poetry to uv. diff --git a/mkdocs.yml b/mkdocs.yml index 1c11fe8114..5392005878 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,6 +35,8 @@ nav: - Wayback: modules/wayback.md - Custom YARA Rules: modules/custom_yara_rules.md - Lightfuzz: modules/lightfuzz.md + - Migration: + - 2.x → 3.0 Breaking Changes: migration/3.0_breaking_changes.md - Misc: - Contribution: contribution.md - Release History: release_history.md From f8403919736d4a6e883664fd84a339f387602051 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Fri, 22 May 2026 17:02:44 -0400 Subject: [PATCH 02/29] drop generic_ssrf claim from migration doc --- docs/migration/3.0_breaking_changes.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index 7ba0a44095..833da2ee59 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -164,8 +164,7 @@ truth. - `webbrute` / `webbrute_shortnames` — ffuf replacements, also via blasthttp. - `bucket_hetzner`, `shodan_enterprise`, `trajan`, `legba`. - Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq`. -- Lightfuzz submodules `esi` and `ssrf` (the old `generic_ssrf` module was - deleted in favor of the lightfuzz submodule). +- Lightfuzz submodules `esi` and `ssrf`. ### Module API From 7229b2b4556de7be88a7740d25c237a55e7395e3 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 3 Jun 2026 15:43:19 -0400 Subject: [PATCH 03/29] Note preset/config validation and ${env:} removal --- docs/migration/3.0_breaking_changes.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index 833da2ee59..89b36a4c2c 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -114,6 +114,13 @@ The top-level scan options that used to be implicit are now required to live under `config:` inside a preset (this was previously documented behavior, now enforced and called out in `defaults.yml`). +Presets and `-c key=value` options are now **validated when loaded**: unknown +keys, typos, and wrong value types are rejected up front (usually with a +closest-match hint, e.g. `Did you mean "scope.strict"?`) instead of being +silently ignored. The old `${env:VAR}` interpolation inside config values is +gone too; inject secrets with shell expansion +(`-c modules.shodan.api_key="$SHODAN_KEY"`) or keep them in `secrets.yml`. + --- ## Modules From b4985d392967d8b266f7c4650709c559eaa90a7f Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Tue, 9 Jun 2026 20:16:45 -0400 Subject: [PATCH 04/29] Update blasthttp to >=0.8.0, add http_proxy_exclude --- docs/migration/3.0_breaking_changes.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index 89b36a4c2c..e66e6db934 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -289,6 +289,8 @@ lookups are also wired through the new `bbot_io_api_key` config (or - `web.user_agent_suffix` — appended to the user agent (previously buried as a hidden CLI flag). - `web.http_rate_limit` — global rps cap across the shared blasthttp client. +- `web.http_proxy_exclude` — hosts/CIDRs to exclude from the HTTP proxy + (`NO_PROXY` equivalent). - `web.body_spill.{enabled,cache_mb,compress}` — disk-spill HTTP response bodies to keep them off the Python heap. - `dns.cache_size` — DNS LRU size. @@ -316,7 +318,7 @@ architecture was deleted. `request_batch` / `request_custom_batch` / `curl` methods were replaced by `request()`, `request_batch_stream(urls, threads=10, **kwargs)`, and `download()`. -- The blasthttp dependency line is `blasthttp>=0.7.0`. +- The blasthttp dependency line is `blasthttp>=0.8.0`. Modules that previously instantiated their own `httpx.AsyncClient` or built custom curl invocations must switch to `self.helpers.request(...)` / @@ -350,7 +352,7 @@ custom curl invocations must switch to `self.helpers.request(...)` / - **Lockstep deps**: - `radixtarget >=4.0.1,<5` (composition pattern, no longer subclassed) - `cloudcheck >=10.0.0,<11` - - `blasthttp >=0.7.0` (new) + - `blasthttp >=0.8.0` (new) - `blastdns >=1.9.0,<2` (new) - `asndb >=1.0.4` (new) - `zstandard` (new; used by HTTP body spill) From cfb4dc3e25e55dcb6490c3a44af82cf0829f2542 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 15 Jun 2026 19:12:33 -0400 Subject: [PATCH 05/29] Update migration doc: ssl_verify split, blasthttp 0.9.0, wildcard detection --- docs/migration/3.0_breaking_changes.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index e66e6db934..553a35e093 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -188,6 +188,10 @@ truth. `False` otherwise. Override explicitly if you want different behavior. - `default_discovery_context` now uses `{event.pretty_string}` instead of `{event.data}` — the latter is now a dict for URL-like events (see below). +- New `BaseModule._is_http_wildcard_host(event)` helper. Returns `True` when + the target responds identically to two random paths (catch-all / SPA + router). Used by `webbrute`, `lightfuzz`, and `paramminer` to skip hosts + that would produce only false positives. --- @@ -276,6 +280,7 @@ lookups are also wired through the new `bbot_io_api_key` config (or | `web.httpx_timeout` | `web.blasthttp_timeout` | | `web.httpx_retries` | `web.blasthttp_retries` | | `dns.threads` (global) | `dns.threads` (now per-resolver; default lowered from 25 → 10) | +| `web.ssl_verify` | split into `web.ssl_verify_target` (default `false`) and `web.ssl_verify_infrastructure` (default `true`) | ### Removed @@ -318,7 +323,7 @@ architecture was deleted. `request_batch` / `request_custom_batch` / `curl` methods were replaced by `request()`, `request_batch_stream(urls, threads=10, **kwargs)`, and `download()`. -- The blasthttp dependency line is `blasthttp>=0.8.0`. +- The blasthttp dependency line is `blasthttp>=0.9.0`. Modules that previously instantiated their own `httpx.AsyncClient` or built custom curl invocations must switch to `self.helpers.request(...)` / @@ -352,7 +357,7 @@ custom curl invocations must switch to `self.helpers.request(...)` / - **Lockstep deps**: - `radixtarget >=4.0.1,<5` (composition pattern, no longer subclassed) - `cloudcheck >=10.0.0,<11` - - `blasthttp >=0.8.0` (new) + - `blasthttp >=0.9.0` (new) - `blastdns >=1.9.0,<2` (new) - `asndb >=1.0.4` (new) - `zstandard` (new; used by HTTP body spill) @@ -388,7 +393,8 @@ subclasses, or `bbot.db.sql.models` will need to be ported. - [ ] Stop mutating `event._resolved_hosts` / `event.dns_children` directly; use the new `add_resolved_host` / `add_dns_child` helpers. - [ ] Update config keys: `web.httpx_timeout → web.blasthttp_timeout`, - `web.httpx_retries → web.blasthttp_retries`. + `web.httpx_retries → web.blasthttp_retries`, + `web.ssl_verify → web.ssl_verify_target` / `web.ssl_verify_infrastructure`. - [ ] Update imports: `bbot.db.sql.models → bbot.models.sql`. - [ ] If you ship a custom module that watches URLs but doesn't want blasthttp auto-enabled, set `_disable_auto_module_deps = True`. From 7b62d63f108774f0b011577aa18247f6af9c8540 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Tue, 16 Jun 2026 13:11:29 -0400 Subject: [PATCH 06/29] Update scanning docs: consolidate targets/seeds/blacklists, add ASN + USERNAME types --- docs/scanning/index.md | 157 +++++++++++++++++++++-------------------- 1 file changed, 82 insertions(+), 75 deletions(-) diff --git a/docs/scanning/index.md b/docs/scanning/index.md index df69dcef90..ea11427bcd 100644 --- a/docs/scanning/index.md +++ b/docs/scanning/index.md @@ -13,9 +13,17 @@ bbot -t evilcorp.com -f subdomain-enum -m gowitness -n my_scan -o . If you reuse a scan name, BBOT will automatically append to your previous output files. -## Targets (`-t`) +## Targets (`-t`), Seeds (`-s`), and Blacklists (`-b`) -Targets declare what's in-scope, and seed a scan with initial data. BBOT accepts an unlimited number of targets. They can be any of the following: +BBOT uses three related concepts to control scope and drive a scan: + +- **Targets (`-t`)** define what is **in-scope**. Active modules (e.g. `nuclei`, `portscan`) will only touch targets and their children. +- **Seeds (`-s`)** are the starting data that gets fed into modules. If you don't specify `-s`, **your targets are automatically used as seeds**. +- **Blacklists (`-b`)** define what is **never touched**. Anything matching the blacklist is excluded, even if it would otherwise be in-scope. + +### Accepted Input Types + +Targets, seeds, and blacklists all accept the same input types: - `DNS_NAME` (`evilcorp.com`) - `IP_ADDRESS` (`1.2.3.4`) @@ -23,14 +31,17 @@ Targets declare what's in-scope, and seed a scan with initial data. BBOT accepts - `OPEN_TCP_PORT` (`192.168.0.1:80`) - `URL` (`https://www.evilcorp.com`) - `EMAIL_ADDRESS` (`bob@evilcorp.com`) +- `ASN` (`ASN:17178` or `AS17178`) +- `USERNAME` (`USER:bobsmith`) - `ORG_STUB` (`ORG:evilcorp`) -- `USER_STUB` (`USER:bobsmith`) - `FILESYSTEM` (`FILESYSTEM:/tmp/asdf`) - `MOBILE_APP` (`MOBILE_APP:https://play.google.com/store/apps/details?id=com.evilcorp.app`) +Blacklists additionally accept **regex patterns** prefixed with `RE:` (see [Blacklist by Regex](#blacklist-by-regex)). + Note that BBOT only discriminates down to the host level. This means, for example, if you specify a URL `https://www.evilcorp.com` as the target, the scan will be *seeded* with that URL, but the scope of the scan will be the entire host, `www.evilcorp.com`. Other ports/URLs on that same host may also be scanned. -You can specify targets directly on the command line, load them from files, or both! For example: +You can specify inputs directly on the command line, load them from files, or both: ```bash $ cat targets.txt @@ -45,7 +56,71 @@ https://www.evilcorp.co.uk $ bbot -t targets.txt fsociety.com 5.6.7.0/24 -m portscan ``` -On start, BBOT automatically converts Targets into [Events](events.md). +On start, BBOT automatically converts these inputs into [Events](events.md). + +### Why Separate Targets and Seeds? + +Separating targets from seeds lets you keep a tight scope while still allowing passive discovery outside of it. When BBOT discovers something out-of-scope via a seed, it will still report it, but active modules won't touch it. + +For example, say your target has subdomains that resolve both inside and outside an IP range that defines your scope. You can set the IP range as the **target** and the domain as a **seed**: + +```bash +bbot -t 192.168.1.0/24 -s evilcorp.com -f subdomain-enum -m nuclei +``` + +Any discovered `evilcorp.com` subdomains that resolve within `192.168.1.0/24` will be actively scanned by Nuclei. Others will be discovered and reported, but not touched by active modules. + +### Blacklists + +`-b` / `--blacklist` takes ultimate precedence. Anything in the blacklist is completely excluded from the scan, even if it would otherwise be in-scope based on your targets or seeds. + +```bash +# Scan evilcorp.com, but exclude internal.evilcorp.com and its children +bbot -t evilcorp.com -b internal.evilcorp.com -f subdomain-enum -m portscan nuclei +``` + +#### Blacklist by Regex + +Blacklists also accept regex patterns. These regexes are checked against the full URL, including the host and path. + +To specify a regex, prefix the pattern with `RE:`. For example, to exclude all events containing "signout": + +```bash +bbot -t evilcorp.com -b "RE:signout" +``` + +Note that this would blacklist both of the following events: + +- `[URL] http://evilcorp.com/signout.aspx` +- `[DNS_NAME] signout.evilcorp.com` + +If you only want to blacklist the URL, you could narrow the regex like so: + +```bash +bbot -t evilcorp.com -b 'RE:signout\.aspx$' +``` + +Similar to targets, blacklists can be specified in your preset. The `spider` preset makes use of this to prevent the spider from following logout links: + +```yaml title="spider.yml" +description: Recursive web spider + +modules: + - http + +blacklist: + # Prevent spider from invalidating sessions by logging out + - "RE:/.*(sign|log)[_-]?out" + +config: + web: + # how many links to follow in a row + spider_distance: 2 + # don't follow links whose directory depth is higher than 4 + spider_depth: 4 + # maximum number of links to follow per page + spider_links_per_page: 25 +``` ## Modules (`-m`) @@ -154,7 +229,7 @@ For details on how Ansible playbooks are attached to BBOT modules, see [How to W For pentesters and bug bounty hunters, staying in scope is extremely important. BBOT takes this seriously, meaning that active modules (e.g. `nuclei`) will only touch in-scope resources. -By default, scope is whatever you specify with `-t`. This includes child subdomains. For example, if you specify `-t evilcorp.com`, all its subdomains (`www.evilcorp.com`, `mail.evilcorp.com`, etc.) also become in-scope. +As described [above](#targets-t-seeds-s-and-blacklists-b), targets (`-t`) define what is in-scope and blacklists (`-b`) define what is excluded. Scope includes child subdomains by default -- for example, `-t evilcorp.com` puts `www.evilcorp.com`, `mail.evilcorp.com`, etc. in-scope automatically. ### Scope Distance @@ -179,74 +254,6 @@ If you want to scan **_only_** that specific target hostname and none of its chi Note that `--strict-scope` only applies to targets, but not blacklists. This means that if you put `internal.evilcorp.com` in your blacklist, you can be sure none of its subdomains will be scanned, even when using `--strict-scope`. -### Targets, Seeds, and Blacklists - -BBOT uses three related concepts to control scope and how a scan is driven: - -- **Targets (`-t` / `--targets`)**: Define what is in-scope. These also act as scan seeds if seeds aren't explicitly defined. -- **Seeds (`-s` / `--seeds`)**: Seeds define the starting point for the scan. They drive **passive** modules and can be outside of the explicit target list (out of scope) for those passive modules. If you don’t specify `--seeds`, BBOT will automatically use your targets as seeds. -- **Blacklists (`-b` / `--blacklist`)**: Define what is **never** touched. Anything matching the blacklist is excluded from the scan, even if it would otherwise be in-scope. - -This separation lets you, for example, keep a tight target list for what’s considered in-scope, while still allowing passive modules to discover new subdomains that may ultimately be in-scope. The blacklist helps to mask-off anything that you know should not be scanned. - -For example, lets say you have a target with subdomains that resolve both within, and outside of an IP range that defines your scope. You can set the IP range as the **target**, and then safely let BBOT explore the domain defined in **seeds**. Any discovered assets that are in your scope will automatically be assessed by active modules. - -```bash -bbot -t 192.168.1.0/24 -s evilcorp.com -f subdomain-enum -m nuclei -``` -In this example, any discovered `evilcorp.com` subdomains that resolve within `192.168.1.0/24` will be scanned by `Nuclei`. Any others will be discovered, but not touched by active modules. - -#### Blacklists - -`--blacklist` takes ultimate precedence. Anything in the blacklist is completely excluded from the scan, even if it would otherwise be in-scope based on your targets or seeds. - -```bash -# Scan evilcorp.com, but exclude internal.evilcorp.com and its children -bbot -t evilcorp.com --blacklist internal.evilcorp.com -f subdomain-enum -m portscan nuclei -``` - -#### Blacklist by Regex - -Blacklists also accept regex patterns. These regexes are are checked against the full URL, including the host and path. - -To specify a regex, prefix the pattern with `RE:`. For example, to exclude all events containing "signout", you could do: - -```bash -bbot -t evilcorp.com --blacklist "RE:signout" -``` - -Note that this would blacklist both of the following events: - -- `[URL] http://evilcorp.com/signout.aspx` -- `[DNS_NAME] signout.evilcorp.com` - -If you only want to blacklist the URL, you could narrow the regex like so: - -```bash -bbot -t evilcorp.com --blacklist 'RE:signout\.aspx$' -``` - -Similar to targets, blacklists can be specified in your preset. The `spider` preset makes use of this to prevent the spider from following logout links: - -```yaml title="spider.yml" -description: Recursive web spider - -modules: - - http - -blacklist: - # Prevent spider from invalidating sessions by logging out - - "RE:/.*(sign|log)[_-]?out" - -config: - web: - # how many links to follow in a row - spider_distance: 2 - # don't follow links whose directory depth is higher than 4 - spider_depth: 4 - # maximum number of links to follow per page - spider_links_per_page: 25 -``` ## DNS Wildcards @@ -281,4 +288,4 @@ dns: wildcard_tests: 20 ``` -If that doesn't work you can consider [blacklisting](#targets-seeds-and-blacklists) the offending domain. +If that doesn't work you can consider [blacklisting](#blacklists) the offending domain. From 0471e49e70b01cf8a5e572ade9a9721ae6b08b1e Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Tue, 16 Jun 2026 15:23:05 -0400 Subject: [PATCH 07/29] Update scanning docs: presets, events, output, tips, config, navigation links --- docs/scanning/advanced.md | 2 ++ docs/scanning/configuration.md | 6 ++-- docs/scanning/events.md | 32 +++++++++++-------- docs/scanning/index.md | 2 ++ docs/scanning/output.md | 55 ++++++++++++++++++++++++++++++-- docs/scanning/presets.md | 50 ++++++++++++++++++++++++++--- docs/scanning/tips_and_tricks.md | 47 +++++++++++++-------------- 7 files changed, 145 insertions(+), 49 deletions(-) diff --git a/docs/scanning/advanced.md b/docs/scanning/advanced.md index 9b40ce81cc..3aa08c9ad2 100644 --- a/docs/scanning/advanced.md +++ b/docs/scanning/advanced.md @@ -179,3 +179,5 @@ EXAMPLES ``` <!-- END BBOT HELP OUTPUT --> + +[Next Up: Configuration -->](./configuration.md){ .md-button .md-button--primary } diff --git a/docs/scanning/configuration.md b/docs/scanning/configuration.md index 11d1232023..ef2f680956 100644 --- a/docs/scanning/configuration.md +++ b/docs/scanning/configuration.md @@ -2,7 +2,7 @@ Normally, [Presets](presets.md) are used to configure a scan. However, there may be cases where you want to change BBOT's global defaults so a certain option is always set, even if it's not specified in a preset. -BBOT has a YAML config at `~/.config/bbot.yml`. This is the first config that BBOT loads, so it's a good place to put default settings like `http_proxy`, `max_threads`, or `http_user_agent`. You can also put any module settings here, including **API keys**. +BBOT has a YAML config at `~/.config/bbot/bbot.yml`. This is the first config that BBOT loads, so it's a good place to put default settings like `http_proxy`, `max_threads`, or `http_user_agent`. You can also put any module settings here, including **API keys**. For a list of all possible config options, see: @@ -32,7 +32,7 @@ bbot -t evilcorp.com -c http_proxy=http://127.0.0.1:8080 Or, in `~/.config/bbot/bbot.yml`: -```yaml title="~/.bbot/config/bbot.yml" +```yaml title="~/.config/bbot/bbot.yml" http_proxy: http://127.0.0.1:8080 ``` @@ -41,7 +41,7 @@ These two are equivalent. Config options specified via the command-line take precedence over all others. You can give BBOT a custom config file with `-c myconf.yml`, or individual arguments like this: `-c modules.shodan_dns.api_key=deadbeef`. To display the full and current BBOT config, including any command-line arguments, use `bbot -c`. Note that placing the following in `bbot.yml`: -```yaml title="~/.bbot/config/bbot.yml" +```yaml title="~/.config/bbot/bbot.yml" modules: shodan_dns: api_key: deadbeef diff --git a/docs/scanning/events.md b/docs/scanning/events.md index 44c3aa1c9f..6f1687b382 100644 --- a/docs/scanning/events.md +++ b/docs/scanning/events.md @@ -8,6 +8,17 @@ An Event is a piece of data discovered by BBOT. Examples include `IP_ADDRESS`, ` event type event data source module tags ``` +## Findings + +All vulnerability discoveries, security-relevant observations, and other notable results in BBOT are emitted as **`FINDING`** events. + +Each finding has a **severity** and a **confidence**: + +* **Severity** indicates impact: `INFO`, `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL` +* **Confidence** indicates how certain the finding is: `CONFIRMED`, `HIGH`, `MEDIUM`, `LOW`, or `UNKNOWN` + +Together, these let you quickly prioritize results -- e.g. a `CRITICAL` severity with `CONFIRMED` confidence is immediately actionable, while a `MEDIUM` severity with `LOW` confidence may need manual verification. + ## Event Attributes Each BBOT event has the following attributes. Not all of these attributes are visible in the terminal output. However, they are always saved in `output.json` in the scan output folder. If you want to see them on the terminal, you can use `--json`. @@ -31,6 +42,7 @@ Each BBOT event has the following attributes. Not all of these attributes are vi - `.tags`: a list of tags describing the event (e.g. `mx-record`, `http-title`, etc.) - `.module`: the module that discovered the event - `.module_sequence`: the recent sequence of modules that were executed to discover the event (including omitted events) +- `.host_metadata`: cloud provider, ASN, and other metadata about the host (when available) - `.discovery_context`: a description of the context in which the event was discovered - `.discovery_path`: a list of every discovery context leading to this event - `.parent_chain`: a list of every event UUID leading to the discovery of this event (corresponds exactly to `.discovery_path`) @@ -43,6 +55,7 @@ These attributes allow us to construct a visual graph of events (e.g. in [Neo4j] "id": "DNS_NAME:33bc005c2bdfea4d73e07db733bd11861cf6520e", "uuid": "DNS_NAME:6c96d512-090a-47f0-82e4-6860e46aac13", "scope_description": "in-scope", + "netloc": "link.evilcorp.com", "data": "link.evilcorp.com", "host": "link.evilcorp.com", "resolved_hosts": [ @@ -66,16 +79,16 @@ These attributes allow us to construct a visual graph of events (e.g. in [Neo4j] "web_spider_distance": 0, "scope_distance": 0, "scan": "SCAN:b6ef48bc036bc8d001595ae5061846a7e6beadb6", - "timestamp": "2024-10-18T15:40:13.716880+00:00", + "timestamp": 1729266013.71688, "parent": "DNS_NAME:94c92b7eaed431b37ae2a757fec4e678cc3bd213", "parent_uuid": "DNS_NAME:c737dffa-d4f0-4b6e-a72d-cc8c05bd892e", "tags": [ - "subdomain", "a-record", + "aaaa-record", "cdn-akamai", - "in-scope", "cname-record", - "aaaa-record" + "in-scope", + "subdomain" ], "module": "speculate", "module_sequence": "speculate->speculate", @@ -140,13 +153,4 @@ Below is a full list of event types along with which modules produce/consume the | WEB_PARAMETER | 7 | 4 | hunt, lightfuzz, paramminer_cookies, paramminer_getparams, paramminer_headers, reflected_parameters, web_parameters | excavate, paramminer_cookies, paramminer_getparams, paramminer_headers | <!-- END BBOT EVENTS --> -## Findings - -All vulnerability discoveries, security-relevant observations, and other notable results in BBOT are emitted as **`FINDING`** events. - -Each finding has a **severity** and a **confidence**: - -* **Severity** indicates impact: `INFO`, `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL` -* **Confidence** indicates how certain the finding is: `CONFIRMED`, `HIGH`, `MEDIUM`, `LOW`, or `UNKNOWN` - -Together, these let you quickly prioritize results -- e.g. a `CRITICAL` severity with `CONFIRMED` confidence is immediately actionable, while a `MEDIUM` severity with `LOW` confidence may need manual verification. +[Next Up: Output -->](./output.md){ .md-button .md-button--primary } diff --git a/docs/scanning/index.md b/docs/scanning/index.md index ea11427bcd..f9c735b5bb 100644 --- a/docs/scanning/index.md +++ b/docs/scanning/index.md @@ -289,3 +289,5 @@ dns: ``` If that doesn't work you can consider [blacklisting](#blacklists) the offending domain. + +[Next Up: Presets -->](./presets.md){ .md-button .md-button--primary } diff --git a/docs/scanning/output.md b/docs/scanning/output.md index 463e19e45b..fed9c8c0b2 100644 --- a/docs/scanning/output.md +++ b/docs/scanning/output.md @@ -298,7 +298,7 @@ mail.evilcorp.com portal.evilcorp.com ``` -## Neo4j +### Neo4j Neo4j is the funnest (and prettiest) way to view and interact with BBOT data. @@ -371,6 +371,55 @@ This is not an exhaustive list of clauses, filters, or other means to use cypher Additional note: these sample queries are dependent on the existence of the data in the target neo4j database. +### Emails + +The `emails` output module writes any email addresses found belonging to the target domain to a file (`emails.txt` by default). + +### Web Report + +The `web_report` output module generates a markdown report of web assets, including URLs, technologies, and findings. + +### Nmap XML + +The `nmap_xml` output module exports open ports, DNS names, IP addresses, and protocols in Nmap XML format for compatibility with tools that consume Nmap output. + +### Message Queues + +BBOT supports streaming events as JSON to several message queue systems: + +| Module | Description | Key Config | +|--------|-------------|------------| +| `kafka` | Publish to a Kafka topic | `bootstrap_servers`, `topic` | +| `rabbitmq` | Publish to a RabbitMQ queue | `url`, `queue` | +| `nats` | Publish to a NATS subject | `servers`, `subject` | +| `zeromq` | Publish to a ZeroMQ PUB socket | `zmq_address` | +| `websocket` | Stream to a WebSocket endpoint | `url`, `token` | + +These can be enabled like any other output module: + +```bash +bbot -t evilcorp.com -om kafka -c modules.kafka.bootstrap_servers=localhost:9092 modules.kafka.topic=bbot_events +``` + +### MongoDB + +The `mongo` output module sends events to a MongoDB database. + +```yaml title="mongo_preset.yml" +output_modules: + - mongo + +config: + modules: + mongo: + uri: mongodb://localhost:27017 + database: bbot +``` + +### Python API + +The `python` output module is used when running BBOT via the Python API. It enables programmatic access to events as they are produced. See the [Developer Documentation](../dev/index.md) for details. + ### Web_parameters The `web_parameters` output module will utilize BBOT web parameter extraction capabilities, and output the resulting parameters to a file (web_parameters.txt, by default). Web parameter extraction is disabled by default, but will automatically be enabled when a module is included that consumes WEB_PARAMETER events (including the `web_parameters` output module itself). @@ -379,4 +428,6 @@ This can be useful for those who want to discover new common web parameters or t ```bash bbot -t evilcorp.com -m paramminer_getparams -c modules.paramminer_getparams.wordlist=/path/to/your/new/wordlist.txt -``` \ No newline at end of file +``` + +[Next Up: Tips and Tricks -->](./tips_and_tricks.md){ .md-button .md-button--primary } \ No newline at end of file diff --git a/docs/scanning/presets.md b/docs/scanning/presets.md index 76e0514d68..87068f4a7e 100644 --- a/docs/scanning/presets.md +++ b/docs/scanning/presets.md @@ -25,7 +25,7 @@ output_modules: ## How to use Presets (`-p`) -BBOT has a ready-made collection of presets for common tasks like subdomain enumeration and web spidering. They live in `~/.bbot/presets`. +BBOT ships with a collection of presets for common tasks like subdomain enumeration and web spidering. The defaults live in `bbot/presets` inside the installed package. You can also place custom presets anywhere and reference them by path. To list them, you can do: @@ -111,15 +111,53 @@ bbot -t evilcorp.com -p ./my_spider.yml spider bbot -t evilcorp.com -p spider ./my_spider.yml ``` -## Validating Presets +## Preset Validation -To make sure BBOT is configured the way you expect, you can always check the `--current-preset` to show the final version of the config that will be used when BBOT executes: +BBOT automatically validates presets when they load. If your preset has a typo in a top-level key, an unknown module name, or an invalid config option, BBOT will catch it and suggest the closest match: + +```text +$ bbot -p ./mypreset.yml +ERROR [preset:modlues] Could not find preset option "modlues". Did you mean "modules"? +``` + +This also applies to module config and flags -- for example, misspelling a module name under `config.modules` or using a flag that doesn't exist will produce a helpful error. + +To inspect the final merged preset that BBOT will use (after all includes and overrides are applied), use `--current-preset`: ```bash -# verify the preset is what you want +# show the final resolved preset bbot -p ./mypreset.yml --current-preset + +# show the full config including defaults +bbot -p ./mypreset.yml --current-preset-full ``` +## Preset YAML Reference + +Here is the full list of supported top-level keys in a preset YAML file: + +| Key | Type | Description | +|-----|------|-------------| +| `target` (or `targets`) | list | In-scope targets (see [Accepted Input Types](index.md#accepted-input-types)) | +| `seeds` | list | Seed events to feed into modules. If omitted, targets are used as seeds | +| `blacklist` | list | Excluded targets. Takes ultimate precedence | +| `modules` | list | Scan modules to enable | +| `output_modules` | list | Output modules (default: `csv`, `human`, `json`) | +| `exclude_modules` | list | Modules to exclude | +| `flags` | list | Enable all modules with these flags | +| `require_flags` | list | Only enable modules that have these flags | +| `exclude_flags` | list | Exclude modules that have any of these flags | +| `config` | dict | Config overrides (global and per-module) | +| `include` (or `presets`) | list | Other presets to include | +| `module_dirs` | list | Additional directories to load modules from | +| `conditions` | list | Jinja2 conditions evaluated before scan start | +| `scan_name` | string | Custom scan name (default: random, e.g. `demonic_jimmy`) | +| `output_dir` | string | Custom output directory (default: `~/.bbot`) | +| `description` | string | Human-readable description of the preset | +| `verbose` | bool | Enable verbose logging | +| `debug` | bool | Enable debug logging | +| `silent` | bool | Silence all stderr output | + ## Advanced Usage BBOT Presets support advanced features like file-based targets, custom modules, and custom conditions. @@ -149,7 +187,7 @@ You can mix file paths and literal targets in the same list. If an entry doesn't If you want to use a custom BBOT `.py` module, you can either move it into `bbot/modules` where BBOT is installed, or add its parent folder to `module_dirs` like so: ```yaml title="custom_modules.yml" -# load extra BBOT modules from this locaation +# load extra BBOT modules from this location module_dirs: - /home/user/custom_modules ``` @@ -193,3 +231,5 @@ Conditions use [Jinja](https://palletsprojects.com/p/jinja/), which means they c - `abort(message)` - abort the scan with an optional message If you aren't able to accomplish what you want with conditions, or if you need access to a new variable/function, please let us know on [Github](https://github.com/blacklanternsecurity/bbot/issues/new/choose). + +[Next Up: Events -->](./events.md){ .md-button .md-button--primary } diff --git a/docs/scanning/tips_and_tricks.md b/docs/scanning/tips_and_tricks.md index bf920911fc..f64c0972a7 100644 --- a/docs/scanning/tips_and_tricks.md +++ b/docs/scanning/tips_and_tricks.md @@ -62,48 +62,45 @@ Copy this to `/etc/resolv.conf` (or append the `nameserver` lines to your existi ### Web Spider -The web spider is great for finding juicy data like subdomains, email addresses, and javascript secrets buried in webpages. However since it can lengthen the duration of a scan, it's disabled by default. To enable the web spider, you must increase the value of `web.spider_distance`. +The web spider is great for finding juicy data like subdomains, email addresses, and javascript secrets buried in webpages. However since it can lengthen the duration of a scan, it's disabled by default. To enable it, use one of the built-in spider presets: -The web spider is controlled with three config values: +- **`spider`** -- follows links up to distance 2, depth 4, 25 links per page. Includes a blacklist to avoid following logout links. +- **`spider-heavy`** -- more aggressive: distance 4, depth 6, 50 links per page. -- `web.spider_depth` (default: `1`: the maximum directory depth allowed. This is to prevent the spider from delving too deep into a website. -- `web.spider_distance` (`0` == all spidering disabled, default: `0`): the maximum number of links that can be followed in a row. This is designed to limit the spider in cases where `web.spider_depth` fails (e.g. for an ecommerce website with thousands of base-level URLs). -- `web.spider_links_per_page` (default: `25`): the maximum number of links per page that can be followed. This is designed to save you in cases where a single page has hundreds or thousands of links. +```bash +# spider www.evilcorp.com +bbot -t www.evilcorp.com -p spider -Here is a typical example: +# pair with subdomain enumeration +bbot -t evilcorp.com -p subdomain-enum spider -```yaml title="spider.yml" -config: - web: - spider_depth: 2 - spider_distance: 2 - spider_links_per_page: 25 +# use the heavier spider +bbot -t evilcorp.com -p subdomain-enum spider-heavy ``` -```bash -# run the web spider against www.evilcorp.com -bbot -t www.evilcorp.com -m http -c spider.yml -``` +If you need custom settings, the spider is controlled with three config values: -You can also pair the web spider with subdomain enumeration: +- `web.spider_distance` (`0` == disabled, default: `0`): the maximum number of links that can be followed in a row. +- `web.spider_depth` (default: `1`): the maximum directory depth allowed. +- `web.spider_links_per_page` (default: `25`): the maximum number of links per page that can be followed. ```bash -# spider every subdomain of evilcorp.com -bbot -t evilcorp.com -f subdomain-enum -c spider.yml +# custom spider settings on the command line +bbot -t www.evilcorp.com -m http -c web.spider_distance=3 web.spider_depth=5 ``` ### Exclude CDNs from Port Scan -Use `--exclude-cdns` to filter out unwanted open ports from CDNs and WAFs, e.g. Cloudflare. You can also customize the criteria by setting `modules.portfilter.cdn_tags`. By default, only open ports with `cdn-*` tags are filtered, but you can include all cloud providers by setting `cdn_tags` to `cdn,cloud`: +Use `--exclude-cdn` to filter out unwanted open ports from CDNs and WAFs, e.g. Cloudflare. You can also customize the criteria by setting `modules.portfilter.cdn_tags`. By default, only open ports with `cdn-*` tags are filtered, but you can include all cloud providers by setting `cdn_tags` to `cdn,cloud`: ```bash -bbot -t evilcorp.com --exclude-cdns -c modules.portfilter.cdn_tags=cdn,cloud +bbot -t evilcorp.com --exclude-cdn -c modules.portfilter.cdn_tags=cdn,cloud ``` Additionally, you can customize the allowed ports by setting `modules.portscan.allowed_cdn_ports`. ```bash -bbot -t evilcorp.com --exclude-cdns -c modules.portfilter.allowed_cdn_ports=80,443,8443 +bbot -t evilcorp.com --exclude-cdn -c modules.portfilter.allowed_cdn_ports=80,443,8443 ``` Example preset: @@ -157,9 +154,7 @@ If you have a ready list of hosts/urls and just want to scan them as fast as pos --8<-- "bbot/presets/fast.yml" ``` -If you already have a list of discovered targets (e.g. URLs), you can speed up the scan by skipping BBOT's DNS resolution. You can do this by setting `dns.disable` to `true`: - -If you don't care about DNS-based scope checks, you can go even further by completely disabling DNS resolution: +If you already have a list of discovered targets (e.g. URLs) and don't need DNS-based scope checks, you can go further by completely disabling DNS resolution: ~~~bash # completely disable DNS resolution @@ -197,3 +192,5 @@ bbot -t evilcorp.com -m sslcert -c modules.sslcert.module_threads=50 ``` `module_threads` is one of several [universal module options](./configuration.md) that can be applied to any module. + +[Next Up: Advanced Usage -->](./advanced.md){ .md-button .md-button--primary } From 9bbe8bfa112e3eb1f01405cea53816f1fe338149 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 16:12:58 -0400 Subject: [PATCH 08/29] Rewrite contribution guide, add CONTRIBUTIONS.md repo pointer --- CONTRIBUTIONS.md | 5 +++++ docs/contribution.md | 53 +++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 55 insertions(+), 3 deletions(-) create mode 100644 CONTRIBUTIONS.md diff --git a/CONTRIBUTIONS.md b/CONTRIBUTIONS.md new file mode 100644 index 0000000000..59d24e3985 --- /dev/null +++ b/CONTRIBUTIONS.md @@ -0,0 +1,5 @@ +# Contributing to BBOT + +See our full contribution guide at: + +**https://www.blacklanternsecurity.com/bbot/Dev/contribution/** diff --git a/docs/contribution.md b/docs/contribution.md index b291cea68a..f1d2e3bc9f 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -1,8 +1,55 @@ -# Contribution +# Contributing to BBOT -We welcome contributions! If you have an idea for a new module, or are a Python developer who wants to get involved, please fork us or come talk to us on [Discord](https://discord.com/invite/PZqkgxu5SA). +We welcome contributions! There are a number of ways to contribute to BBOT. -To get started devving, see the following links: +If you want to chat about BBOT or get help with a contribution, come find us on [Discord](https://discord.com/invite/PZqkgxu5SA). + +## Open an Issue + +Opening an issue is the simplest way to contribute. If you spot a bug or see a way to improve something, open an issue and we'll do our best to address it promptly. + +## Open a Discussion + +If you have an idea for a new module or feature, or just have questions about existing modules, discussions are the correct place for it. Issues opened for these items will most likely be moved here anyway. + +## Pull Requests + +We love pull requests. Before you submit them though, there are a few things to discuss. + +### Should It Be an Issue Instead? + +As AI-assisted contributions have become more common, we've seen an increase in large PRs trying to change things that touch complex core internals. Much of the core BBOT code is extremely complicated and load-bearing. If your AI doesn't grasp the full context, untangling your PR is going to take us longer than if we made the changes ourselves. + +Basically, don't feel pressured to submit a fix just because you found a bug. If the fix isn't straightforward, an issue that shows clearly how to reproduce the problem is often more helpful than a sprawling PR. + +### AI Policy + +Modern AI is an incredibly valuable tool for development. We definitely use it. However, it is still just a tool, and it REALLY matters how you use it. + +- **Blind trust is not good.** Even the most advanced models make horrific mistakes, just as they have moments of brilliance. +- **Human accountability.** If your name is on the code, it's up to you to understand it. A simple rule: don't submit anything you don't understand. +- **No low quality submissions.** If you just submitted the same slop PR to 20 other repos, we are going to close it. If it ignores all the normal patterns in use in BBOT and does its own thing, we are going to close it. If you make a good-faith attempt at doing it right, we're more than happy to help you along. + +We provide an [AGENTS.md](https://github.com/blacklanternsecurity/bbot/blob/dev/AGENTS.md) file in the repo root. Feed this to your LLM before working on BBOT -- it describes our conventions and should help steer AI-assisted contributions in the right direction. + +### Other Advice + +- The pull requests we are most likely to approve are ones that address a very specific thing in a focused way. Don't let that stop you from writing a whole new, even complicated module. However, expect some critical feedback and some rounds of revisions. +- Be mindful of security. We have had multiple CVEs, including critical CVEs. If your module is doing potentially dangerous things -- running commands, reading and writing files, etc. -- expect a lot of extra scrutiny. + +### Tests + +We believe tests are the backbone of any large scale project. Think of them like a save-point for a capability. Without them, regressions constantly creep in and accumulate. + +Every module MUST have module tests. There are tons of examples to draw from with existing modules. We can also help you write the tests. They shouldn't be there just to be there -- they should really exercise as much of the code in your module as possible. For more details, see [Unit Tests](./dev/tests.md). + +## Contributor License Agreement + +Like many open-source projects, we ask that you sign our [Contributor License Agreement](https://github.com/blacklanternsecurity/CLA/blob/main/ICLA.md) before we can accept your contribution. + +## Development + +To get started developing, see the following links: - [Setting up a Dev Environment](./dev/dev_environment.md) - [How to Write a BBOT Module](./dev/module_howto.md) From 5539196fe890dd500b4952996980eb6fded311a2 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 16:30:01 -0400 Subject: [PATCH 09/29] Add dev branch guidance, fix typo in contribution guide --- AGENTS.md | 4 ++-- docs/contribution.md | 16 ++++++++++++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 055fbcca78..6c34ed62d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,8 +76,8 @@ ruff format --check # verify formatting without changes ### Git Workflow - `stable` - production releases -- `dev` - active development, PR target -- Feature branches are created from `dev` +- `dev` - active development, **almost all PRs should target this branch** +- Feature branches should be created from `dev` --- diff --git a/docs/contribution.md b/docs/contribution.md index f1d2e3bc9f..5d4015a3f8 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -29,19 +29,27 @@ Modern AI is an incredibly valuable tool for development. We definitely use it. - **Blind trust is not good.** Even the most advanced models make horrific mistakes, just as they have moments of brilliance. - **Human accountability.** If your name is on the code, it's up to you to understand it. A simple rule: don't submit anything you don't understand. - **No low quality submissions.** If you just submitted the same slop PR to 20 other repos, we are going to close it. If it ignores all the normal patterns in use in BBOT and does its own thing, we are going to close it. If you make a good-faith attempt at doing it right, we're more than happy to help you along. +- **Don't let the AI edit tests.** Of course, there will be some times where a test edit is legitimately needed, but this is a common antipattern for AI agents. -We provide an [AGENTS.md](https://github.com/blacklanternsecurity/bbot/blob/dev/AGENTS.md) file in the repo root. Feed this to your LLM before working on BBOT -- it describes our conventions and should help steer AI-assisted contributions in the right direction. +We don't require AI disclosure, but it's not discouraged either (usually we can tell though). + +We provide an [AGENTS.md](https://github.com/blacklanternsecurity/bbot/blob/dev/AGENTS.md) file in the repo root. Feed this to your LLM before working on BBOT. It describes our conventions and should help steer AI-assisted contributions in the right direction. ### Other Advice +- **Work off the `dev` branch.** Almost all PRs should target `dev`, not `stable`. Create your feature branch from `dev` and open your PR against it. - The pull requests we are most likely to approve are ones that address a very specific thing in a focused way. Don't let that stop you from writing a whole new, even complicated module. However, expect some critical feedback and some rounds of revisions. -- Be mindful of security. We have had multiple CVEs, including critical CVEs. If your module is doing potentially dangerous things -- running commands, reading and writing files, etc. -- expect a lot of extra scrutiny. +- Be mindful of security. We have had multiple CVEs, including critical CVEs. If your module is doing potentially dangerous things like running commands, reading and writing files, etc. - expect a lot of extra scrutiny. ### Tests We believe tests are the backbone of any large scale project. Think of them like a save-point for a capability. Without them, regressions constantly creep in and accumulate. -Every module MUST have module tests. There are tons of examples to draw from with existing modules. We can also help you write the tests. They shouldn't be there just to be there -- they should really exercise as much of the code in your module as possible. For more details, see [Unit Tests](./dev/tests.md). +Every module MUST have module tests. There are tons of examples to draw from with existing modules. We can also help you write the tests. They shouldn't be there just to be there, they should really exercise as much of the code in your module as possible. + +If you make a change to any core code, helpers, etc - chances are if you break anything, our massive amount of tests (as of this writing's creation over 300) will catch it. So run them. Try to run the entire test suite against your code before you submit it. + +For more details, see [Unit Tests](./dev/tests.md). ## Contributor License Agreement @@ -49,7 +57,7 @@ Like many open-source projects, we ask that you sign our [Contributor License Ag ## Development -To get started developing, see the following links: +Ready to dive in? See the following links to get started: - [Setting up a Dev Environment](./dev/dev_environment.md) - [How to Write a BBOT Module](./dev/module_howto.md) From f0780bcea1de615f1ebcfe83bf319f4b85fa7fc6 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 16:39:12 -0400 Subject: [PATCH 10/29] Add 2.8.1-2.8.6 to release history, fix duplicate 1.0.5 --- docs/release_history.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/release_history.md b/docs/release_history.md index 90349d6c4f..3e4776af4e 100644 --- a/docs/release_history.md +++ b/docs/release_history.md @@ -1,5 +1,8 @@ +### 2.8.6 - Jun 16, 2026 +- 2.8.6 is expected to be the last 2.x release. Supported Python versions: 3.10-3.13. +- [https://github.com/blacklanternsecurity/bbot/pull/3199](https://github.com/blacklanternsecurity/bbot/pull/3199) + ### 2.8.5 - Jun 16, 2026 -- 2.8.5 is expected to be the last 2.x release. Supported Python versions: 3.10-3.13. - [https://github.com/blacklanternsecurity/bbot/pull/3185](https://github.com/blacklanternsecurity/bbot/pull/3185) ### 2.8.4 - Mar 17, 2026 @@ -86,6 +89,3 @@ ### 1.0.5 - Mar 10, 2023 - [https://github.com/blacklanternsecurity/bbot/pull/352](https://github.com/blacklanternsecurity/bbot/pull/352) - -### 1.0.5 - Mar 10, 2023 -- [https://github.com/blacklanternsecurity/bbot/pull/352](https://github.com/blacklanternsecurity/bbot/pull/352) From 9ec17978c39dbfcddc1d79f67ca3b517412a4bb9 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 16:45:49 -0400 Subject: [PATCH 11/29] Fix dev reference page, update troubleshooting --- docs/dev/index.md | 14 ++++++-------- docs/troubleshooting.md | 27 +++++++++++++++++++-------- 2 files changed, 25 insertions(+), 16 deletions(-) diff --git a/docs/dev/index.md b/docs/dev/index.md index 699c5799c8..d9ed3e8792 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -49,18 +49,16 @@ For a full listing of `Scanner` attributes and functions, see the [`Scanner` Cod You can specify any number of targets: ```python -# create a scan against multiple targets scan = Scanner( "evilcorp.com", "evilcorp.org", - "evilcorp.ce", "4.3.2.1", "1.2.3.4/24", presets=["subdomain-enum"] ) # this is the same as: -targets = ["evilcorp.com", "evilcorp.org", "evilcorp.ce", "4.3.2.1", "1.2.3.4/24"] +targets = ["evilcorp.com", "evilcorp.org", "4.3.2.1", "1.2.3.4/24"] scan = Scanner(*targets, presets=["subdomain-enum"]) ``` @@ -68,18 +66,18 @@ For more details, including which types of targets are valid, see [Targets](../s #### Other Custom Options -In many cases, using a [Preset](../scanning/presets.md) like `subdomain-enum` is sufficient. However, the `Scanner` is flexible and accepts many other arguments that can override the default functionality. You can specify [`flags`](../scanning/index.md#flags-f), [`modules`](../scanning/index.md#modules-m), [`output_modules`](../output.md), a [target list / `seeds` / `blacklist`](../scanning/index.md#targets-seeds-and-blacklists), and custom [`config` options](../scanning/configuration.md): +In many cases, using a [Preset](../scanning/presets.md) like `subdomain-enum` is sufficient. However, the `Scanner` is flexible and accepts many other arguments that can override the default functionality. You can specify [`flags`](../scanning/index.md#flags-f), [`modules`](../scanning/index.md#modules-m), [`output_modules`](../scanning/output.md), a [target list / `seeds` / `blacklist`](../scanning/index.md#targets-t-seeds-s-and-blacklists-b), and custom [`config` options](../scanning/configuration.md): ```python -# create a scan against multiple targets scan = Scanner( - # targets + # targets (positional args define scope) "evilcorp.com", + "evilcorp.org", "4.3.2.1", # enable these presets presets=["subdomain-enum"], - # explicitly define in-scope targets - target=["evilcorp.com", "evilcorp.org"], + # seeds drive passive modules without affecting scope + seeds=["1.2.3.4/24"], # blacklist these hosts blacklist=["prod.evilcorp.com"], # also enable these individual modules diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d3be020ad8..502fe0dc1b 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,22 +1,22 @@ # Troubleshooting -## Installation troubleshooting +## Installation Troubleshooting - `Fatal error from pip prevented installation.` - `ERROR: No matching distribution found for bbot` -- `bash: /home/user/.local/bin/bbot: /home/user/.local/pipx/venvs/bbot/bin/python: bad interpreter` -If you get errors resembling any of the above, it's probably because your Python version is too old. To install a newer version (3.10+ is required), you will need to do something like this: +If you get errors like the above, it's probably because your Python version is too old. BBOT requires Python 3.10+. + ```bash # install a newer version of python -sudo apt install python3.10 python3.10-venv +sudo apt install python3.12 python3.12-venv # install pipx -python3.10 -m pip install --user pipx +python3.12 -m pip install --user pipx # add pipx to your path -python3.10 -m pipx ensurepath +python3.12 -m pipx ensurepath # reboot reboot # install bbot -python3.10 -m pipx install bbot +python3.12 -m pipx install bbot # run bbot bbot --help ``` @@ -24,8 +24,19 @@ bbot --help ## `ModuleNotFoundError` If you run into a `ModuleNotFoundError`, try running your `bbot` command again with `--force-deps`. This will repair your modules' Python dependencies. +## Clear BBOT Cache +BBOT caches module data, wordlists, and other resources under `~/.bbot`. After an upgrade, stale cache files can sometimes cause unexpected errors. If you're seeing strange behavior after updating, try clearing it: + +```bash +# remove the BBOT cache directory +rm -rf ~/.bbot + +# BBOT will recreate it on the next run +bbot --help +``` + ## Regenerate Config -As a troubleshooting step it is sometimes useful to clear out your older configs and let BBOT generate new ones. This will ensure that new defaults are property restored, etc. +As a troubleshooting step it is sometimes useful to clear out your older configs and let BBOT generate new ones. This will ensure that new defaults are properly restored, etc. ```bash # make a backup of the old configs mv ~/.config/bbot ~/.config/bbot.bak From b184270ae7bb4a680af29acbdf432a63d4161995 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 16:46:35 -0400 Subject: [PATCH 12/29] Fix dev environment setup: git URL, dev branch, ruff check --- docs/dev/dev_environment.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/dev/dev_environment.md b/docs/dev/dev_environment.md index 94e72abd73..355a0c6ed1 100644 --- a/docs/dev/dev_environment.md +++ b/docs/dev/dev_environment.md @@ -11,15 +11,19 @@ The following will show you how to set up a fully functioning python environment ```bash # clone your forked repo and cd into it -git clone git@github.com/<username>/bbot.git +git clone git@github.com:<username>/bbot.git cd bbot -# install uv +# switch to the dev branch and create a feature branch +git checkout dev +git checkout -b my-feature + +# install uv (if you haven't already) curl -LsSf https://astral.sh/uv/install.sh | sh # install pip dependencies uv sync --group dev -# install pre-commit hooks, etc. +# install pre-commit hooks uv run pre-commit install # enter virtual environment @@ -32,7 +36,8 @@ bbot --help - After making your changes, run the tests locally to ensure they pass. ```bash -# auto-format code indentation, etc. +# lint and auto-format +ruff check ruff format # run tests From c34ca1a30606870c4d692ce6bc2d112bbfc69427 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 16:53:59 -0400 Subject: [PATCH 13/29] Update module howto and architecture docs for current codebase --- docs/dev/architecture.md | 2 +- docs/dev/module_howto.md | 63 ++++++++++++++++++++++------------------ 2 files changed, 35 insertions(+), 30 deletions(-) diff --git a/docs/dev/architecture.md b/docs/dev/architecture.md index a2547154f5..bd23c13b32 100644 --- a/docs/dev/architecture.md +++ b/docs/dev/architecture.md @@ -6,7 +6,7 @@ Here is a basic overview of BBOT's internal architecture. Being both ***recursive*** and ***event-driven***, BBOT makes heavy use of queues. These enable smooth communication between the modules, and ensure that large numbers of events can be produced without slowing down or clogging up the scan. -Every module in BBOT has both an ***incoming*** and ***outgoing*** queue. Event types matching the module's `WATCHED_EVENTS` (e.g. `DNS_NAME`) are queued in its incoming queue, and processed by the module's `handle_event()` (or `handle_batch()` in the case of batched modules). If the module finds anything interesting, it creates an event and places it in its outgoing queue, to be processed by the scan and redistributed to other modules. +Every module in BBOT has both an ***incoming*** and ***outgoing*** queue. Event types matching the module's `watched_events` (e.g. `DNS_NAME`) are queued in its incoming queue, and processed by the module's `handle_event()` (or `handle_batch()` in the case of batched modules). If the module finds anything interesting, it creates an event and places it in its outgoing queue, to be processed by the scan and redistributed to other modules. ## Event Flow diff --git a/docs/dev/module_howto.md b/docs/dev/module_howto.md index 1b67d982fd..ec68850544 100644 --- a/docs/dev/module_howto.md +++ b/docs/dev/module_howto.md @@ -10,21 +10,24 @@ Here we'll go over a basic example of writing a custom BBOT module. - the class must have the same name as your file (case-insensitive) 1. Define in `watched_events` what type of data your module will consume 1. Define in `produced_events` what type of data your module will produce -1. Define (via `flags`) whether your module is `active` or `passive`, and optionally whether it's `loud` or `invasive` +1. Define (via `flags`) whether your module is `active` or `passive`, and whether it's `safe`, `loud`, or `invasive` 1. **Put your main logic in `.handle_event()`** Here is an example of a simple module that performs whois lookups: ```python title="bbot/modules/whois.py" from bbot.modules.base import BaseModule +from bbot.core.config.models import BaseModuleConfig, Field class whois(BaseModule): watched_events = ["DNS_NAME"] # watch for DNS_NAME events - produced_events = ["WHOIS"] # we produce WHOIS events - flags = ["passive"] - meta = {"description": "Query WhoisXMLAPI for WHOIS data"} - options = {"api_key": ""} # module config options - options_desc = {"api_key": "WhoisXMLAPI Key"} + produced_events = ["DNS_NAME"] # we produce DNS_NAME events + flags = ["passive", "safe"] + meta = {"description": "Query WhoisXMLAPI for related domains"} + + class Config(BaseModuleConfig): + api_key: str = Field("", description="WhoisXMLAPI Key", sensitive=True, mandatory=True) + per_domain_only = True # only run once per domain base_url = "https://www.whoisxmlapi.com/whoisserver/WhoisService" @@ -43,7 +46,8 @@ class whois(BaseModule): self.hugeinfo(f"Visiting {url}") response = await self.helpers.request(url) if response is not None: - await self.emit_event(response.json(), "WHOIS", parent=event) + for related_domain in response.json().get("domains", []): + await self.emit_event(related_domain, "DNS_NAME", parent=event) ``` ## Test your new module @@ -76,9 +80,9 @@ For details on how tests are written, see [Unit Tests](./tests.md). ## `handle_event()` and `emit_event()` -The `handle_event()` method is the most important part of the module. By overriding this method, you control what the module does. During a scan, when an [event](./scanning/events.md) from your `watched_events` is encountered (a `DNS_NAME` in this example), `handle_event()` is automatically called with that event as its argument. +The `handle_event()` method is the most important part of the module. By overriding this method, you control what the module does. During a scan, when an [event](../scanning/events.md) from your `watched_events` is encountered (a `DNS_NAME` in this example), `handle_event()` is automatically called with that event as its argument. -The `emit_event()` method is how modules return data. When you call `emit_event()`, it creates an [event](./scanning/events.md) and outputs it, sending it any modules that are interested in that data type. +The `emit_event()` method is how modules return data. When you call `emit_event()`, it creates an [event](../scanning/events.md) and outputs it, sending it any modules that are interested in that data type. ## `setup_deps()` and `setup()` @@ -115,43 +119,44 @@ async def setup(self): ## Module Config Options -Each module can have its own set of config options. These live in the `options` and `options_desc` attributes on your class. Both are dictionaries; `options` is for defaults and `options_desc` is for descriptions. Here is a typical example: +Each module can have its own set of config options. These are defined as a `Config` inner class that inherits from `BaseModuleConfig`, using pydantic `Field` for defaults and descriptions. Here is a typical example: -```python title="bbot/modules/nmap.py" -class nmap(BaseModule): +```python title="bbot/modules/portscan.py" +from bbot.core.config.models import BaseModuleConfig, Field + +class portscan(BaseModule): # ... - options = { - "top_ports": 100, - "ports": "", - "timing": "T4", - "skip_host_discovery": True, - } - options_desc = { - "top_ports": "Top ports to scan (default 100) (to override, specify 'ports')", - "ports": "Ports to scan", - "timing": "-T<0-5>: Set timing template (higher is faster)", - "skip_host_discovery": "skip host discovery (-Pn)", - } + class Config(BaseModuleConfig): + top_ports: int = Field(100, description="Top ports to scan (default 100) (to override, specify 'ports')") + ports: str = Field("", description="Ports to scan") + rate: int = Field(300, description="Rate in packets per second") + wait: int = Field(5, description="Seconds to wait for replies after scan is complete") async def setup(self): self.ports = self.config.get("ports", "") - self.timing = self.config.get("timing", "T4") + self.rate = self.config.get("rate", 300) self.top_ports = self.config.get("top_ports", 100) - self.skip_host_discovery = self.config.get("skip_host_discovery", True) return True ``` -Once you've defined these variables, you can pass the options via `-c`: +For API keys and other secrets, use `sensitive=True` (redacted in logs) and `mandatory=True` (module soft-fails if not set): + +```python +class Config(BaseModuleConfig): + api_key: str = Field("", description="API Key", sensitive=True, mandatory=True) +``` + +Once you've defined these fields, you can pass the options via `-c`: ```bash -bbot -m nmap -c modules.nmap.top_ports=250 +bbot -m portscan -c modules.portscan.top_ports=250 ``` ... or via the config: ```yaml title="~/.config/bbot/bbot.yml" modules: - nmap: + portscan: top_ports: 250 ``` From 5a2ec9317ad2bed5641115905f6b523d0ebbd6f1 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 17:12:23 -0400 Subject: [PATCH 14/29] Update dev docs: tests, core deps page, architecture, discord bot example --- docs/dev/core_dependencies.md | 77 ++++++++++++++++++++++ docs/dev/tests.md | 121 ++++++++++++++++++++++++++++++---- examples/discord_bot.py | 38 +++++++---- mkdocs.yml | 1 + 4 files changed, 211 insertions(+), 26 deletions(-) create mode 100644 docs/dev/core_dependencies.md diff --git a/docs/dev/core_dependencies.md b/docs/dev/core_dependencies.md new file mode 100644 index 0000000000..50781bbcee --- /dev/null +++ b/docs/dev/core_dependencies.md @@ -0,0 +1,77 @@ +# Core Dependencies + +BBOT is built on top of several purpose-built libraries, most of which are maintained by [Black Lantern Security](https://github.com/blacklanternsecurity). Understanding these will help you navigate the codebase and avoid reinventing things that already exist. + +Libraries marked with **BLS** are maintained by us. + +## blasthttp (BLS) + +[blasthttp](https://github.com/blacklanternsecurity/blasthttp) is a Rust-backed HTTP client built for speed. It handles all of BBOT's outbound HTTP traffic, including module API calls, web spidering, and brute-forcing. + +Module authors don't typically interact with blasthttp directly. Instead, use `self.helpers.request()`: + +```python +response = await self.helpers.request("https://example.com/api") +if response and response.status_code == 200: + data = response.json() +``` + +The shared blasthttp client is accessible at `self.helpers.blasthttp` and respects the global `web.http_rate_limit` config. + +## blastdns (BLS) + +[blastdns](https://github.com/blacklanternsecurity/blastdns) is a Rust-backed async DNS resolver. It powers all of BBOT's DNS resolution with built-in caching, retries, and per-resolver parallelism. BBOT spins up multiple workers per resolver in `/etc/resolv.conf`, so adding more resolvers directly speeds up scans. + +Accessed via `self.helpers.dns`: + +```python +# resolve a hostname +results = await self.helpers.resolve("example.com", rdtype="A") + +# resolve with full record details +results = await self.helpers.resolve_full("example.com", rdtype="CNAME") +``` + +## cloudcheck (BLS) + +[cloudcheck](https://github.com/blacklanternsecurity/cloudcheck) identifies which cloud provider (if any) owns a given IP or domain. It maintains a regularly-updated database of IP ranges for AWS, Azure, GCP, Cloudflare, and dozens of other providers. BBOT uses it to tag events with cloud provider info (e.g. `cloud-aws`, `cdn-cloudflare`). + +Accessed via `self.helpers.cloudcheck`: + +```python +result = await self.helpers.cloudcheck.lookup("1.2.3.4") +# CloudCheckResult with provider name, type (cdn, cloud, waf, etc.) +``` + +## radixtarget (BLS) + +[radixtarget](https://github.com/blacklanternsecurity/radixtarget) is a high-performance radix tree for IP/DNS lookups. It is the data structure behind BBOT's target, seed, and blacklist storage, enabling fast scope checks against large target lists with CIDR and subdomain matching. + +```python +from radixtarget import RadixTarget + +rt = RadixTarget() +rt.insert("10.0.0.0/8") +rt.search("10.1.2.3") # True +``` + +## asndb (BLS) + +[asndb](https://github.com/blacklanternsecurity/asndb) provides ASN lookups for IP addresses, returning the owning organization, AS number, and subnet. BBOT uses it to enrich events with ASN metadata and to support ASN targets (e.g. `bbot -t AS1234`). + +Accessed via `self.helpers.asn`: + +```python +result = await self.helpers.asn.lookup("8.8.8.8") +# {"asn": 15169, "name": "GOOGLE", "description": "Google LLC", ...} +``` + +## Other Key Dependencies + +| Library | What BBOT uses it for | +|---------|----------------------| +| [pydantic](https://docs.pydantic.dev/) | Preset/config validation, module config schemas (`BaseModuleConfig`) | +| [ansible-runner](https://ansible-runner.readthedocs.io/) | Module dependency installation (`deps_apt`, `deps_ansible`) | +| [yara-python](https://yara.readthedocs.io/) | Pattern matching in `excavate` (secret/credential extraction from HTTP responses) | +| [deepdiff](https://zepworks.com/deepdiff/) | HTTP response comparison for web modules (baseline diffing, wildcard detection) | +| [dnspython](https://dnspython.readthedocs.io/) | DNS record type constants and utilities (blastdns handles actual resolution) | diff --git a/docs/dev/tests.md b/docs/dev/tests.md index 1013ea85db..f5c6105c05 100644 --- a/docs/dev/tests.md +++ b/docs/dev/tests.md @@ -54,7 +54,7 @@ class TestMyModule(ModuleTestBase): async def setup_after_prep(self, module_test): # mock HTTP response module_test.blasthttp_mock.add_response( - url="https://api.com/sudomains?apikey=deadbeef&domain=blacklanternsecurity.com", + url="https://api.com/subdomains?apikey=deadbeef&domain=blacklanternsecurity.com", json={ "subdomains": [ "www.blacklanternsecurity.com", @@ -81,25 +81,122 @@ class TestMyModule(ModuleTestBase): assert "dev.blacklanternsecurity.com" in dns_names, "failed to find subdomain #2" ``` +### Mocking HTTP responses + +`module_test.blasthttp_mock` intercepts all outbound HTTP requests during tests. Requests to `127.0.0.1`/`localhost` pass through to the real test HTTP server, but everything else is mocked. + +```python + async def setup_after_prep(self, module_test): + # JSON response + module_test.blasthttp_mock.add_response( + url="https://api.example.com/lookup?domain=blacklanternsecurity.com", + json={"subdomains": ["www.blacklanternsecurity.com"]}, + ) + + # plain text response + module_test.blasthttp_mock.add_response( + url="https://example.com/data.txt", + text="some plain text response", + ) + + # error response + module_test.blasthttp_mock.add_response( + url="https://example.com/broken", + status_code=500, + text="Internal Server Error", + ) + + # match on specific method and headers + module_test.blasthttp_mock.add_response( + url="https://api.example.com/submit", + method="POST", + match_headers={"Authorization": "Bearer mytoken"}, + json={"status": "ok"}, + ) +``` + +### Mocking DNS + +Use `module_test.mock_dns()` to control DNS resolution. Supports A, AAAA, CNAME, MX, TXT, and other record types: + +```python + async def setup_after_prep(self, module_test): + await module_test.mock_dns({ + "blacklanternsecurity.com": {"A": ["127.0.0.88"]}, + "www.blacklanternsecurity.com": {"CNAME": ["blacklanternsecurity.com"]}, + "mail.blacklanternsecurity.com": {"MX": ["mx.example.com"]}, + }) +``` + ### Debugging a test -Similar to debugging from within a module, you can debug from within a test using `self.log.critical()`, etc: +You can debug from within a test using standard Python logging via `self.log`: ```python def check(self, module_test, events): for e in events: - # bright red - self.log.critical(e.type) - # bright green - self.log.hugesuccess(e.data) - # bright orange - self.log.hugewarning(e.tags) - # bright blue - self.log.hugeinfo(e.parent) + self.log.critical(e.type) # bright red + self.log.warning(e.tags) # orange + self.log.info(e.data) # blue + self.log.debug(e.parent) # grey (requires -d) +``` + +### Advanced test features + +#### HTTP request handlers + +For dynamic HTTP responses, use `set_expect_requests_handler` with a custom handler function: + +```python +import re +from werkzeug.wrappers import Response + +class TestMyModule(ModuleTestBase): + targets = ["http://127.0.0.1:8888"] + modules_overrides = ["http", "mymodule"] + + def request_handler(self, request): + if request.path == "/api/data": + return Response('{"results": ["found"]}', status=200) + return Response("Not Found", status=404) + + async def setup_before_prep(self, module_test): + module_test.set_expect_requests_handler( + expect_args=re.compile("/.*"), + request_handler=self.request_handler, + ) +``` + +#### OOB interaction mocking + +For modules that use out-of-band interactions (interactsh): + +```python + async def setup_before_prep(self, module_test): + self.interactsh_mock_instance = module_test.mock_interactsh("mymodule") + + from bbot.core.helpers.helper import ConfigAwareHelper + module_test.monkeypatch.setattr( + ConfigAwareHelper, "interactsh", + lambda *a, **kw: self.interactsh_mock_instance, + ) +``` + +#### Other class attributes + +```python +class TestMyModule(ModuleTestBase): + targets = ["http://127.0.0.1:8888"] # override default target + modules_overrides = ["http", "mymodule"] # control which modules are enabled + module_name = "mymodule" # if your class name doesn't match the module + config_overrides = {"modules": {"mymodule": {"option": "value"}}} + blacklist = ["bad.example.com"] # set a blacklist + seeds = ["seed.example.com"] # set seeds separate from targets + skip_distro_tests = True # skip when running in CI distro tests (e.g. docker-dependent tests) ``` -### More advanced tests +### More examples If you have questions about tests or need to write a more advanced test, come talk to us on [GitHub](https://github.com/blacklanternsecurity/bbot/discussions) or [Discord](https://discord.com/invite/PZqkgxu5SA). -It's also a good idea to look through our [existing tests](https://github.com/blacklanternsecurity/bbot/tree/stable/bbot/test/test_step_2/module_tests). BBOT has over a hundred of them, so you might find one that's similar to what you're trying to do. +It's also a good idea to look through our [existing tests](https://github.com/blacklanternsecurity/bbot/tree/dev/bbot/test/test_step_2/module_tests). BBOT has over 300 of them, so you might find one that's similar to what you're trying to do. diff --git a/examples/discord_bot.py b/examples/discord_bot.py index f435b0301c..b953dc3307 100644 --- a/examples/discord_bot.py +++ b/examples/discord_bot.py @@ -2,7 +2,6 @@ from discord.ext import commands from bbot.scanner import Scanner -from bbot.modules.output.discord import Discord class BBOTDiscordBot(commands.Cog): @@ -14,13 +13,13 @@ class BBOTDiscordBot(commands.Cog): 2. Create a new application 3. Create an invite link for the bot, visit the link to invite it to your server - Your Application --> OAuth2 --> URL Generator - - For Scopes, select "bot"" + - For Scopes, select "bot" - For Bot Permissions, select: - Read Messages/View Channels - Send Messages 4. Turn on "Message Content Intent" - Your Application --> Bot --> Privileged Gateway Intents --> Message Content Intent - 5. Copy your Discord Bot Token and put it at the top this file + 5. Copy your Discord Bot Token and put it at the top of this file - Your Application --> Bot --> Reset Token 6. Run this script @@ -36,27 +35,38 @@ def __init__(self): @commands.command(name="scan", description="Scan a target with BBOT.") async def scan(self, ctx, target: str): + # stop any existing scan if self.current_scan is not None: self.current_scan.stop() - await ctx.send(f"Starting scan against {target}.") + await ctx.send(f"Starting scan against **{target}**") - # creates scan instance - self.current_scan = Scanner(target, flags="subdomain-enum") - discord_module = Discord(self.current_scan) + # create scan instance + self.current_scan = Scanner(target, presets=["subdomain-enum"]) - seen = set() + # iterate through results and send them to the channel num_events = 0 - # start scan and iterate through results async for event in self.current_scan.async_start(): - if hash(event) in seen: - continue - seen.add(hash(event)) - await ctx.send(discord_module.format_message(event)) + # format findings differently to show severity + if event.type == "FINDING": + severity = event.data.get("severity", "INFO") + description = event.data.get("description", "") + event_text = f"`[FINDING]` [{severity}] **{description}**" + else: + event_text = f"`[{event.type}]` **`{event.data}`**" + await ctx.send(event_text) num_events += 1 - await ctx.send(f"Finished scan against {target}. {num_events:,} results.") + await ctx.send(f"Finished scan against **{target}**. {num_events:,} results.") self.current_scan = None + @commands.command(name="stop", description="Stop the current scan.") + async def stop(self, ctx): + if self.current_scan is None: + await ctx.send("No scan is currently running.") + return + self.current_scan.stop() + await ctx.send("Scan stopped.") + if __name__ == "__main__": intents = discord.Intents.default() diff --git a/mkdocs.yml b/mkdocs.yml index 1c11fe8114..a7345cb833 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -42,6 +42,7 @@ nav: - Developer Manual: - Development Overview: dev/index.md - Setting Up a Dev Environment: dev/dev_environment.md + - Core Dependencies: dev/core_dependencies.md - BBOT Internal Architecture: dev/architecture.md - How to Write a BBOT Module: dev/module_howto.md - Validating & Inspecting Presets: dev/preset_validation.md From 916cd4e1980d8fa778e6beac634ce3fc4bb0ac26 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 17:14:53 -0400 Subject: [PATCH 15/29] Remove dead engine code, reparent WebError/DNSError to BBOTError --- bbot/core/engine.py | 707 --------------------------- bbot/errors.py | 8 +- bbot/test/test_step_1/test_engine.py | 146 ------ docs/dev/engine.md | 5 - mkdocs.yml | 1 - 5 files changed, 2 insertions(+), 865 deletions(-) delete mode 100644 bbot/core/engine.py delete mode 100644 bbot/test/test_step_1/test_engine.py delete mode 100644 docs/dev/engine.md diff --git a/bbot/core/engine.py b/bbot/core/engine.py deleted file mode 100644 index 7a33f0da71..0000000000 --- a/bbot/core/engine.py +++ /dev/null @@ -1,707 +0,0 @@ -import os -import sys -import zmq -import pickle -import asyncio -import inspect -import logging -import tempfile -import traceback -import contextlib -import contextvars -import zmq.asyncio -import multiprocessing -from pathlib import Path -from concurrent.futures import CancelledError -from contextlib import asynccontextmanager, suppress - -from bbot.core import CORE -from bbot.errors import BBOTEngineError -from bbot.core.helpers.async_helpers import get_event_loop -from bbot.core.multiprocess import SHARED_INTERPRETER_STATE -from bbot.core.helpers.misc import rand_string, in_exception_chain - - -error_sentinel = object() - - -class EngineBase: - """ - Base Engine class for Server and Client. - - An Engine is a simple and lightweight RPC implementation that allows offloading async tasks - to a separate process. It leverages ZeroMQ in a ROUTER-DEALER configuration. - - BBOT makes use of this by spawning a dedicated engine for DNS and HTTP tasks. - This offloads I/O and helps free up the main event loop for other tasks. - - To use Engine, you must subclass both EngineClient and EngineServer. - - See the respective EngineClient and EngineServer classes for usage examples. - """ - - ERROR_CLASS = BBOTEngineError - - def __init__(self, debug=False): - self._shutdown_status = False - self.log = logging.getLogger(f"bbot.core.{self.__class__.__name__.lower()}") - self._engine_debug = debug - - def pickle(self, obj): - try: - return pickle.dumps(obj) - except Exception as e: - self.log.error(f"Error serializing object: {obj}: {e}") - self.log.trace(traceback.format_exc()) - return error_sentinel - - def unpickle(self, binary): - try: - return pickle.loads(binary) - except Exception as e: - self.log.error(f"Error deserializing binary: {e}") - self.log.trace(f"Offending binary: {binary}") - self.log.trace(traceback.format_exc()) - return error_sentinel - - async def _infinite_retry(self, callback, *args, **kwargs): - interval = kwargs.pop("_interval", 300) - context = kwargs.pop("_context", "") - # default overall timeout of 10 minutes (300 second interval * 2 iterations) - max_retries = kwargs.pop("_max_retries", 1) - if not context: - context = f"{callback.__name__}({args}, {kwargs})" - retries = 0 - while not self._shutdown_status: - try: - return await asyncio.wait_for(callback(*args, **kwargs), timeout=interval) - except (TimeoutError, asyncio.exceptions.TimeoutError): - self.log.debug(f"{self.name}: Timeout after {interval:,} seconds {context}, retrying...") - retries += 1 - if max_retries is not None and retries > max_retries: - raise TimeoutError(f"Timed out after {(max_retries + 1) * interval:,} seconds {context}") - - def engine_debug(self, *args, **kwargs): - if self._engine_debug: - self.log.trace(*args, **kwargs) - - -class EngineClient(EngineBase): - """ - The client portion of BBOT's RPC Engine. - - To create an engine, you must create a subclass of this class and also - define methods for each of your desired functions. - - Note that this only supports async functions. If you need to offload a synchronous function to another CPU, use BBOT's multiprocessing pool instead. - - Any CPU or I/O intense logic should be implemented in the EngineServer. - - These functions are typically stubs whose only job is to forward the arguments to the server. - - Functions with the same names should be defined on the EngineServer. - - The EngineClient must specify its associated server class via the `SERVER_CLASS` variable. - - Depending on whether your function is a generator, you will use either `run_and_return()`, or `run_and_yield`. - - Examples: - >>> from bbot.core.engine import EngineClient - >>> - >>> class MyClient(EngineClient): - >>> SERVER_CLASS = MyServer - >>> - >>> async def my_function(self, **kwargs) - >>> return await self.run_and_return("my_function", **kwargs) - >>> - >>> async def my_generator(self, **kwargs): - >>> async for _ in self.run_and_yield("my_generator", **kwargs): - >>> yield _ - """ - - SERVER_CLASS = None - - def __init__(self, debug=False, **kwargs): - self.name = f"EngineClient {self.__class__.__name__}" - super().__init__(debug=debug) - self.process = None - if self.SERVER_CLASS is None: - raise ValueError(f"Must set EngineClient SERVER_CLASS, {self.SERVER_CLASS}") - self.CMDS = dict(self.SERVER_CLASS.CMDS) - for k, v in list(self.CMDS.items()): - self.CMDS[v] = k - self.socket_address = f"zmq_{rand_string(8)}.sock" - self.socket_path = Path(tempfile.gettempdir()) / self.socket_address - self.server_kwargs = kwargs.pop("server_kwargs", {}) - self._server_process = None - self.context = zmq.asyncio.Context() - self.context.setsockopt(zmq.LINGER, 0) - self.sockets = set() - - def check_error(self, message): - if isinstance(message, dict) and len(message) == 1 and "_e" in message: - self.engine_debug(f"{self.name}: got error message: {message}") - error, trace = message["_e"] - error = self.ERROR_CLASS(error) - error.engine_traceback = trace - self.engine_debug(f"{self.name}: raising {error.__class__.__name__}") - raise error - return False - - async def run_and_return(self, command, *args, **kwargs): - fn_str = f"{command}({args}, {kwargs})" - self.engine_debug(f"{self.name}: executing run-and-return {fn_str}") - if self._shutdown_status and not command == "_shutdown": - self.log.verbose(f"{self.name} has been shut down and is not accepting new tasks") - return - async with self.new_socket() as socket: - try: - message = self.make_message(command, args=args, kwargs=kwargs) - if message is error_sentinel: - return - await socket.send(message) - binary = await self._infinite_retry(socket.recv, _context=f"waiting for return value from {fn_str}") - except BaseException: - try: - await self.send_cancel_message(socket, fn_str) - except Exception: - self.log.debug(f"{self.name}: {fn_str} failed to send cancel message after exception") - self.log.trace(traceback.format_exc()) - raise - # self.log.debug(f"{self.name}.{command}({kwargs}) got binary: {binary}") - message = self.unpickle(binary) - self.engine_debug(f"{self.name}: {fn_str} got return value: {message}") - # error handling - if self.check_error(message): - return - return message - - async def run_and_yield(self, command, *args, **kwargs): - fn_str = f"{command}({args}, {kwargs})" - self.engine_debug(f"{self.name}: executing run-and-yield {fn_str}") - if self._shutdown_status: - self.log.verbose("Engine has been shut down and is not accepting new tasks") - return - message = self.make_message(command, args=args, kwargs=kwargs) - if message is error_sentinel: - return - async with self.new_socket() as socket: - # TODO: synchronize server-side generator by limiting qsize - # socket.setsockopt(zmq.RCVHWM, 1) - # socket.setsockopt(zmq.SNDHWM, 1) - await socket.send(message) - while 1: - try: - binary = await self._infinite_retry( - socket.recv, _context=f"waiting for new iteration from {fn_str}" - ) - # self.log.debug(f"{self.name}.{command}({kwargs}) got binary: {binary}") - message = self.unpickle(binary) - self.engine_debug(f"{self.name}: {fn_str} got iteration: {message}") - # error handling - if self.check_error(message) or self.check_stop(message): - break - yield message - except (StopAsyncIteration, GeneratorExit) as e: - exc_name = e.__class__.__name__ - self.engine_debug(f"{self.name}.{command} got {exc_name}") - try: - await self.send_cancel_message(socket, fn_str) - except Exception: - self.engine_debug(f"{self.name}.{command} failed to send cancel message after {exc_name}") - self.log.trace(traceback.format_exc()) - break - - async def send_cancel_message(self, socket, context): - """ - Send a cancel message and wait for confirmation from the server - """ - # -1 == special "cancel" signal - message = pickle.dumps({"c": -1}) - await self._infinite_retry(socket.send, message) - while 1: - response = await self._infinite_retry( - socket.recv, _context=f"waiting for CANCEL_OK from {context}", _max_retries=4 - ) - response = pickle.loads(response) - if isinstance(response, dict): - response = response.get("m", "") - if response == "CANCEL_OK": - break - - async def send_shutdown_message(self): - async with self.new_socket() as socket: - # -99 == special shutdown message - message = pickle.dumps({"c": -99}) - with suppress(TimeoutError, asyncio.exceptions.TimeoutError): - await asyncio.wait_for(socket.send(message), 0.5) - with suppress(TimeoutError, asyncio.exceptions.TimeoutError): - while 1: - response = await asyncio.wait_for(socket.recv(), 0.5) - response = pickle.loads(response) - if isinstance(response, dict): - response = response.get("m", "") - if response == "SHUTDOWN_OK": - break - - def check_stop(self, message): - if isinstance(message, dict) and len(message) == 1 and "_s" in message: - return True - return False - - def make_message(self, command, args=None, kwargs=None): - try: - cmd_id = self.CMDS[command] - except KeyError: - raise KeyError(f'Command "{command}" not found. Available commands: {",".join(self.available_commands)}') - message = {"c": cmd_id} - if args: - message["a"] = args - if kwargs: - message["k"] = kwargs - return pickle.dumps(message) - - @property - def available_commands(self): - return [s for s in self.CMDS if isinstance(s, str)] - - def start_server(self): - process_name = multiprocessing.current_process().name - if SHARED_INTERPRETER_STATE.is_scan_process: - kwargs = dict(self.server_kwargs) - # if we're in tests, we use a single event loop to avoid weird race conditions - # this allows us to more easily mock http, etc. - if os.environ.get("BBOT_TESTING", "") == "True": - kwargs["_loop"] = get_event_loop() - kwargs["debug"] = self._engine_debug - self.process = CORE.create_process( - target=self.server_process, - args=( - self.SERVER_CLASS, - self.socket_path, - ), - kwargs=kwargs, - custom_name=f"BBOT {self.__class__.__name__}", - ) - self.process.start() - return self.process - else: - raise BBOTEngineError( - f"Tried to start server from process {process_name}. Did you forget \"if __name__ == '__main__'?\"" - ) - - @staticmethod - def server_process(server_class, socket_path, **kwargs): - try: - loop = kwargs.pop("_loop", None) - engine_server = server_class(socket_path, **kwargs) - if loop is not None: - future = asyncio.run_coroutine_threadsafe(engine_server.worker(), loop) - future.result() - else: - asyncio.run(engine_server.worker()) - except (asyncio.CancelledError, KeyboardInterrupt, CancelledError): - return - except Exception: - import traceback - - log = logging.getLogger("bbot.core.engine.server") - log.critical(f"Unhandled error in {server_class.__name__} server process: {traceback.format_exc()}") - - @asynccontextmanager - async def new_socket(self): - if self._server_process is None: - self._server_process = self.start_server() - while not self.socket_path.exists(): - self.engine_debug(f"{self.name}: waiting for server process to start...") - await asyncio.sleep(0.1) - socket = self.context.socket(zmq.DEALER) - socket.setsockopt(zmq.LINGER, 0) # Discard pending messages immediately disconnect() or close() - socket.setsockopt(zmq.SNDHWM, 0) # Unlimited send buffer - socket.setsockopt(zmq.RCVHWM, 0) # Unlimited receive buffer - socket.connect(f"ipc://{self.socket_path}") - self.sockets.add(socket) - try: - yield socket - finally: - self.sockets.remove(socket) - with suppress(Exception): - socket.close() - - async def shutdown(self): - if not self._shutdown_status: - self._shutdown_status = True - self.log.verbose(f"{self.name}: shutting down...") - # send shutdown signal - await self.send_shutdown_message() - # then terminate context - try: - self.context.destroy(linger=0) - except Exception: - print(traceback.format_exc(), file=sys.stderr) - try: - self.context.term() - except Exception: - print(traceback.format_exc(), file=sys.stderr) - # terminate the server process/thread - if self._server_process is not None: - try: - self._server_process.join(timeout=5) - if self._server_process.is_alive(): - # threads don't have terminate/kill, only processes do - terminate = getattr(self._server_process, "terminate", None) - if callable(terminate): - terminate() - self._server_process.join(timeout=3) - if self._server_process.is_alive(): - kill = getattr(self._server_process, "kill", None) - if callable(kill): - kill() - except Exception: - with suppress(Exception): - kill = getattr(self._server_process, "kill", None) - if callable(kill): - kill() - self._server_process = None - # delete socket file on exit - self.socket_path.unlink(missing_ok=True) - - -class EngineServer(EngineBase): - """ - The server portion of BBOT's RPC Engine. - - Methods defined here must match the methods in your EngineClient. - - To use the functions, you must create mappings for them in the CMDS attribute, as shown below. - - Examples: - >>> from bbot.core.engine import EngineServer - >>> - >>> class MyServer(EngineServer): - >>> CMDS = { - >>> 0: "my_function", - >>> 1: "my_generator", - >>> } - >>> - >>> def my_function(self, arg1=None): - >>> await asyncio.sleep(1) - >>> return str(arg1) - >>> - >>> def my_generator(self): - >>> for i in range(10): - >>> await asyncio.sleep(1) - >>> yield i - """ - - CMDS = {} - - def __init__(self, socket_path, debug=False): - self.name = f"EngineServer {self.__class__.__name__}" - super().__init__(debug=debug) - self.engine_debug(f"{self.name}: finished setup 1 (_debug={self._engine_debug})") - self.socket_path = socket_path - self.client_id_var = contextvars.ContextVar("client_id", default=None) - # task <--> client id mapping - self.tasks = {} - # child tasks spawned by main tasks - self.child_tasks = {} - self.engine_debug(f"{self.name}: finished setup 2 (_debug={self._engine_debug})") - if self.socket_path is not None: - # create ZeroMQ context - self.context = zmq.asyncio.Context() - # ROUTER socket can handle multiple concurrent requests - self.socket = self.context.socket(zmq.ROUTER) - self.socket.setsockopt(zmq.LINGER, 0) # Discard pending messages immediately disconnect() or close() - self.socket.setsockopt(zmq.SNDHWM, 0) # Unlimited send buffer - self.socket.setsockopt(zmq.RCVHWM, 0) # Unlimited receive buffer - # create socket file - self.socket.bind(f"ipc://{self.socket_path}") - self.engine_debug(f"{self.name}: finished setup 3 (_debug={self._engine_debug})") - - @contextlib.contextmanager - def client_id_context(self, value): - token = self.client_id_var.set(value) - try: - yield - finally: - self.client_id_var.reset(token) - - async def run_and_return(self, client_id, command_fn, *args, **kwargs): - fn_str = f"{command_fn.__name__}({args}, {kwargs})" - self.engine_debug(fn_str) - with self.client_id_context(client_id): - try: - self.engine_debug(f"{self.name}: starting run-and-return {fn_str}") - try: - result = await command_fn(*args, **kwargs) - except BaseException as e: - if in_exception_chain(e, (KeyboardInterrupt, asyncio.CancelledError)): - log_fn = self.log.debug - else: - log_fn = self.log.error - error = f"{self.name}: error in {fn_str}: {e}" - trace = traceback.format_exc() - log_fn(error) - self.log.trace(trace) - result = {"_e": (error, trace)} - finally: - self.tasks.pop(client_id, None) - self.engine_debug(f"{self.name}: sending response to {fn_str}: {result}") - await self.send_socket_multipart(client_id, result) - except BaseException as e: - self.log.critical( - f"Unhandled exception in {self.name}.run_and_return({client_id}, {command_fn}, {args}, {kwargs}): {e}" - ) - self.log.critical(traceback.format_exc()) - finally: - self.engine_debug(f"{self.name} finished run-and-return {fn_str}") - - async def run_and_yield(self, client_id, command_fn, *args, **kwargs): - fn_str = f"{command_fn.__name__}({args}, {kwargs})" - with self.client_id_context(client_id): - try: - self.engine_debug(f"{self.name}: starting run-and-yield {fn_str}") - try: - async for _ in command_fn(*args, **kwargs): - self.engine_debug(f"{self.name}: sending iteration for {fn_str}: {_}") - await self.send_socket_multipart(client_id, _) - except BaseException as e: - if in_exception_chain(e, (KeyboardInterrupt, asyncio.CancelledError)): - log_fn = self.log.debug - else: - log_fn = self.log.error - error = f"{self.name}: error in {fn_str}: {e}" - trace = traceback.format_exc() - log_fn(error) - self.log.trace(trace) - result = {"_e": (error, trace)} - await self.send_socket_multipart(client_id, result) - finally: - self.engine_debug(f"{self.name}: reached end of run-and-yield iteration for {fn_str}") - # _s == special signal that means StopIteration - await self.send_socket_multipart(client_id, {"_s": None}) - self.tasks.pop(client_id, None) - except BaseException as e: - self.log.critical( - f"Unhandled exception in {self.name}.run_and_yield({client_id}, {command_fn}, {args}, {kwargs}): {e}" - ) - self.log.critical(traceback.format_exc()) - finally: - self.engine_debug(f"{self.name}: finished run-and-yield {fn_str}") - - async def send_socket_multipart(self, client_id, message): - try: - message = pickle.dumps(message) - await self._infinite_retry(self.socket.send_multipart, [client_id, message]) - except Exception as e: - self.log.verbose(f"{self.name}: error sending ZMQ message: {e}") - self.log.trace(traceback.format_exc()) - - def check_error(self, message): - if message is error_sentinel: - return True - - async def worker(self): - self.engine_debug(f"{self.name}: starting worker") - try: - while 1: - client_id, binary = await self.socket.recv_multipart() - message = self.unpickle(binary) - self.engine_debug(f"{self.name} got message: {message}") - if self.check_error(message): - continue - - cmd = message.get("c", None) - if not isinstance(cmd, int): - self.log.warning(f"{self.name}: no command sent in message: {message}") - continue - - # -1 == cancel task - if cmd == -1: - self.engine_debug(f"{self.name} got cancel signal") - await self.send_socket_multipart(client_id, {"m": "CANCEL_OK"}) - await self.cancel_task(client_id) - continue - - # -99 == shutdown task - if cmd == -99: - self.log.verbose(f"{self.name} got shutdown signal") - await self.send_socket_multipart(client_id, {"m": "SHUTDOWN_OK"}) - await self._shutdown() - return - - args = message.get("a", ()) - if not isinstance(args, tuple): - self.log.warning(f"{self.name}: received invalid args of type {type(args)}, should be tuple") - continue - kwargs = message.get("k", {}) - if not isinstance(kwargs, dict): - self.log.warning(f"{self.name}: received invalid kwargs of type {type(kwargs)}, should be dict") - continue - - command_name = self.CMDS[cmd] - command_fn = getattr(self, command_name, None) - - if command_fn is None: - self.log.warning(f'{self.name} has no function named "{command_fn}"') - continue - - if inspect.isasyncgenfunction(command_fn): - self.engine_debug(f"{self.name}: creating run-and-yield coroutine for {command_name}()") - coroutine = self.run_and_yield(client_id, command_fn, *args, **kwargs) - else: - self.engine_debug(f"{self.name}: creating run-and-return coroutine for {command_name}()") - coroutine = self.run_and_return(client_id, command_fn, *args, **kwargs) - - self.engine_debug(f"{self.name}: creating task for {command_name}() coroutine") - task = asyncio.create_task(coroutine) - self.tasks[client_id] = task, command_fn, args, kwargs - self.engine_debug(f"{self.name}: finished creating task for {command_name}() coroutine") - except BaseException as e: - await self._shutdown() - if not in_exception_chain(e, (KeyboardInterrupt, asyncio.CancelledError)): - self.log.error(f"{self.name}: error in EngineServer worker: {e}") - self.log.trace(traceback.format_exc()) - finally: - self.engine_debug(f"{self.name}: finished worker()") - - async def _shutdown(self): - if not self._shutdown_status: - self.log.verbose(f"{self.name}: shutting down...") - self._shutdown_status = True - await self.cancel_all_tasks() - context = getattr(self, "context", None) - if context is not None: - try: - context.destroy(linger=0) - except Exception: - self.log.trace(traceback.format_exc()) - try: - context.term() - except Exception: - self.log.trace(traceback.format_exc()) - self.log.verbose(f"{self.name}: finished shutting down") - - async def task_pool(self, fn, args_kwargs, threads=10, timeout=300, global_kwargs=None): - if global_kwargs is None: - global_kwargs = {} - - tasks = {} - args_kwargs = list(args_kwargs) - - def new_task(): - if args_kwargs: - kwargs = {} - tracker = None - args = args_kwargs.pop(0) - if isinstance(args, (list, tuple)): - # you can specify a custom tracker value if you want - # this helps with correlating results - with suppress(ValueError): - args, kwargs, tracker = args - # or you can just specify args/kwargs - with suppress(ValueError): - args, kwargs = args - - if not isinstance(kwargs, dict): - raise ValueError(f"kwargs must be dict (got: {kwargs})") - if not isinstance(args, (list, tuple)): - args = [args] - - task = self.new_child_task(fn(*args, **kwargs, **global_kwargs)) - tasks[task] = (args, kwargs, tracker) - - for _ in range(threads): # Start initial batch of tasks - new_task() - - while tasks: # While there are tasks pending - # Wait for the first task to complete - finished = await self.finished_tasks(tasks, timeout=timeout) - for task in finished: - result = task.result() - (args, kwargs, tracker) = tasks.pop(task) - yield (args, kwargs, tracker), result - new_task() - - def new_child_task(self, coro): - """ - Create a new asyncio task, making sure to track it based on the client id. - - This allows the task to be automatically cancelled if its parent is cancelled. - """ - client_id = self.client_id_var.get() - task = asyncio.create_task(coro) - - if client_id: - - def remove_task(t): - tasks = self.child_tasks.get(client_id, set()) - tasks.discard(t) - if not tasks: - self.child_tasks.pop(client_id, None) - - task.add_done_callback(remove_task) - - try: - self.child_tasks[client_id].add(task) - except KeyError: - self.child_tasks[client_id] = {task} - - return task - - async def finished_tasks(self, tasks, timeout=None): - """ - Given a list of asyncio tasks, return the ones that are finished with an optional timeout - """ - if tasks: - try: - done, _ = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED, timeout=timeout) - return done - except BaseException as e: - if isinstance(e, (TimeoutError, asyncio.exceptions.TimeoutError)): - self.log.warning(f"{self.name}: Timeout after {timeout:,} seconds in finished_tasks({tasks})") - for task in list(tasks): - task.cancel() - self._await_cancelled_task(task) - else: - if not in_exception_chain(e, (KeyboardInterrupt, asyncio.CancelledError)): - self.log.error(f"{self.name}: Unhandled exception in finished_tasks({tasks}): {e}") - self.log.trace(traceback.format_exc()) - raise - return set() - - async def cancel_task(self, client_id): - parent_task = self.tasks.pop(client_id, None) - if parent_task is None: - return - parent_task, _cmd, _args, _kwargs = parent_task - self.engine_debug(f"{self.name}: Cancelling client id {client_id} (task: {parent_task})") - parent_task.cancel() - child_tasks = self.child_tasks.pop(client_id, set()) - if child_tasks: - self.engine_debug(f"{self.name}: Cancelling {len(child_tasks):,} child tasks for client id {client_id}") - for child_task in child_tasks: - child_task.cancel() - - for task in [parent_task] + list(child_tasks): - await self._await_cancelled_task(task) - - async def _await_cancelled_task(self, task): - try: - await asyncio.wait_for(task, timeout=10) - except (TimeoutError, asyncio.exceptions.TimeoutError): - self.log.trace(f"{self.name}: Timeout cancelling task: {task}") - return - except (KeyboardInterrupt, asyncio.CancelledError): - return - except BaseException as e: - self.log.error(f"Unhandled error in {task.get_coro().__name__}(): {e}") - self.log.trace(traceback.format_exc()) - - async def cancel_all_tasks(self): - for client_id in list(self.tasks): - await self.cancel_task(client_id) - for client_id, tasks in self.child_tasks.items(): - for task in list(tasks): - await self._await_cancelled_task(task) diff --git a/bbot/errors.py b/bbot/errors.py index db295da81d..d204222adb 100644 --- a/bbot/errors.py +++ b/bbot/errors.py @@ -62,15 +62,11 @@ class PresetAbortError(PresetConditionError): pass -class BBOTEngineError(BBOTError): +class WebError(BBOTError): pass -class WebError(BBOTEngineError): - pass - - -class DNSError(BBOTEngineError): +class DNSError(BBOTError): pass diff --git a/bbot/test/test_step_1/test_engine.py b/bbot/test/test_step_1/test_engine.py deleted file mode 100644 index 653c3dcd6c..0000000000 --- a/bbot/test/test_step_1/test_engine.py +++ /dev/null @@ -1,146 +0,0 @@ -from ..bbot_fixtures import * - - -@pytest.mark.asyncio -async def test_engine(): - from bbot.core.engine import EngineClient, EngineServer - - counter = 0 - yield_cancelled = False - yield_errored = False - return_started = False - return_finished = False - return_cancelled = False - return_errored = False - - class TestEngineServer(EngineServer): - CMDS = { - 0: "return_thing", - 1: "yield_stuff", - } - - async def return_thing(self, n): - nonlocal return_started - nonlocal return_finished - nonlocal return_cancelled - nonlocal return_errored - try: - return_started = True - await asyncio.sleep(n) - return_finished = True - return f"thing{n}" - except asyncio.CancelledError: - return_cancelled = True - raise - except Exception: - return_errored = True - raise - - async def yield_stuff(self, n): - nonlocal counter - nonlocal yield_cancelled - nonlocal yield_errored - try: - for i in range(n): - yield f"thing{i}" - counter += 1 - await asyncio.sleep(0.1) - except asyncio.CancelledError: - yield_cancelled = True - raise - except Exception: - yield_errored = True - raise - - class TestEngineClient(EngineClient): - SERVER_CLASS = TestEngineServer - - async def return_thing(self, n): - return await self.run_and_return("return_thing", n) - - async def yield_stuff(self, n): - async for _ in self.run_and_yield("yield_stuff", n): - yield _ - - test_engine = TestEngineClient() - - # test return functionality - return_res = await test_engine.return_thing(1) - assert return_res == "thing1" - - # test async generator - assert counter == 0 - assert yield_cancelled is False - yield_res = [r async for r in test_engine.yield_stuff(13)] - assert yield_res == [f"thing{i}" for i in range(13)] - assert len(yield_res) == 13 - assert counter == 13 - - # test async generator with cancellation - counter = 0 - yield_cancelled = False - yield_errored = False - agen = test_engine.yield_stuff(1000) - async for r in agen: - if counter > 10: - await agen.aclose() - break - await asyncio.sleep(5) - assert yield_cancelled is True - assert yield_errored is False - assert counter < 15 - - # test async generator with error - yield_cancelled = False - yield_errored = False - agen = test_engine.yield_stuff(None) - with pytest.raises(BBOTEngineError): - async for _ in agen: - pass - assert yield_cancelled is False - assert yield_errored is True - - # test return with cancellation - return_started = False - return_finished = False - return_cancelled = False - return_errored = False - task = asyncio.create_task(test_engine.return_thing(2)) - await asyncio.sleep(1) - task.cancel() - with pytest.raises(asyncio.CancelledError): - await task - await asyncio.sleep(0.1) - assert return_started is True - assert return_finished is False - assert return_cancelled is True - assert return_errored is False - - # test return with late cancellation - return_started = False - return_finished = False - return_cancelled = False - return_errored = False - task = asyncio.create_task(test_engine.return_thing(1)) - await asyncio.sleep(2) - task.cancel() - result = await task - assert result == "thing1" - assert return_started is True - assert return_finished is True - assert return_cancelled is False - assert return_errored is False - - # test return with error - return_started = False - return_finished = False - return_cancelled = False - return_errored = False - with pytest.raises(BBOTEngineError): - result = await test_engine.return_thing(None) - assert return_started is True - assert return_finished is False - assert return_cancelled is False - assert return_errored is True - - await test_engine.shutdown() diff --git a/docs/dev/engine.md b/docs/dev/engine.md deleted file mode 100644 index d77bd3970e..0000000000 --- a/docs/dev/engine.md +++ /dev/null @@ -1,5 +0,0 @@ -::: bbot.core.engine.EngineBase - -::: bbot.core.engine.EngineClient - -::: bbot.core.engine.EngineServer diff --git a/mkdocs.yml b/mkdocs.yml index a7345cb833..b811be387c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,7 +55,6 @@ nav: - Target: dev/target.md - BaseModule: dev/basemodule.md - BBOTCore: dev/core.md - - Engine: dev/engine.md - Helpers: - Overview: dev/helpers/index.md - Command: dev/helpers/command.md From aa996435b103b0bee6836ffa648a615b3149eb34 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 17:18:31 -0400 Subject: [PATCH 16/29] Fix helper docs: stale DNS methods, syntax error, engine reference --- docs/dev/helpers/command.md | 2 +- docs/dev/helpers/dns.md | 7 ++++--- docs/dev/helpers/index.md | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/dev/helpers/command.md b/docs/dev/helpers/command.md index 3716d2037a..8b433a2e7a 100644 --- a/docs/dev/helpers/command.md +++ b/docs/dev/helpers/command.md @@ -7,7 +7,7 @@ These helpers can be invoked directly from `self.helpers`, but inside a module t ```python # simple subprocess ls_result = await self.run_process("ls", "-l") -for line ls_result.stdout.splitlines(): +for line in ls_result.stdout.splitlines(): # ... # iterate through each line in real time diff --git a/docs/dev/helpers/dns.md b/docs/dev/helpers/dns.md index 5a51d61168..53fc0225c1 100644 --- a/docs/dev/helpers/dns.md +++ b/docs/dev/helpers/dns.md @@ -5,7 +5,7 @@ These are helpers related to DNS resolution. They are used throughout BBOT and i Note that these helpers can be invoked directly from `self.helpers`, e.g.: ```python -self.helpers.resolve("evilcorp.com") +await self.helpers.resolve("evilcorp.com") ``` ::: bbot.core.helpers.dns.DNSHelper @@ -13,7 +13,8 @@ self.helpers.resolve("evilcorp.com") options: members: - resolve - - resolve_batch - - resolve_raw + - resolve_full + - resolve_multi_full + - resolve_batch_full - is_wildcard - is_wildcard_domain diff --git a/docs/dev/helpers/index.md b/docs/dev/helpers/index.md index cc27ed1f2b..545e7e2b01 100644 --- a/docs/dev/helpers/index.md +++ b/docs/dev/helpers/index.md @@ -1,6 +1,6 @@ # BBOT Helpers -In this section are various helper functions that are designed to make your life easier when devving on BBOT. Whether you're extending BBOT by writing a module or working on its core engine, these functions are designed to act as useful machine parts to perform essential tasks, such as making a web request or executing a DNS query. +In this section are various helper functions that are designed to make your life easier when devving on BBOT. Whether you're extending BBOT by writing a module or working on its core, these functions are designed to act as useful machine parts to perform essential tasks, such as making a web request or executing a DNS query. The vast majority of these helpers can be accessed directly from the `.helpers` attribute of a scan or module, like so: From 0ab5e5417233f9494bb72468988309c3807ca3b5 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 17:32:30 -0400 Subject: [PATCH 17/29] Update README: Python 3.10+, blastdns, sync presets, fix TOC --- README.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 5eabdc7eeb..2b4b69602c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ [![bbot_banner](https://github.com/user-attachments/assets/f02804ce-9478-4f1e-ac4d-9cf5620a3214)](https://github.com/blacklanternsecurity/bbot) -[![Python Version](https://img.shields.io/badge/python-3.10+-FF8400)](https://www.python.org) [![License](https://img.shields.io/badge/license-AGPLv3-FF8400.svg)](https://github.com/blacklanternsecurity/bbot/blob/dev/LICENSE) [![DEF CON Recon Village 2024](https://img.shields.io/badge/DEF%20CON%20Demo%20Labs-2023-FF8400.svg)](https://www.reconvillage.org/talks) [![PyPi Downloads](https://static.pepy.tech/personalized-badge/bbot?right_color=orange&left_color=grey)](https://pepy.tech/project/bbot) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) [![Tests](https://github.com/blacklanternsecurity/bbot/actions/workflows/tests.yml/badge.svg?branch=stable)](https://github.com/blacklanternsecurity/bbot/actions?query=workflow%3A"tests") [![Codecov](https://codecov.io/gh/blacklanternsecurity/bbot/branch/dev/graph/badge.svg?token=IR5AZBDM5K)](https://codecov.io/gh/blacklanternsecurity/bbot) [![Discord](https://img.shields.io/discord/859164869970362439)](https://discord.com/invite/PZqkgxu5SA) +[![Python Version](https://img.shields.io/badge/python-3.10+-FF8400)](https://www.python.org) [![License](https://img.shields.io/badge/license-AGPLv3-FF8400.svg)](https://github.com/blacklanternsecurity/bbot/blob/dev/LICENSE) [![PyPi Downloads](https://static.pepy.tech/personalized-badge/bbot?right_color=orange&left_color=grey)](https://pepy.tech/project/bbot) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) [![Tests](https://github.com/blacklanternsecurity/bbot/actions/workflows/tests.yml/badge.svg?branch=stable)](https://github.com/blacklanternsecurity/bbot/actions?query=workflow%3A"tests") [![Codecov](https://codecov.io/gh/blacklanternsecurity/bbot/branch/dev/graph/badge.svg?token=IR5AZBDM5K)](https://codecov.io/gh/blacklanternsecurity/bbot) [![Discord](https://img.shields.io/discord/859164869970362439)](https://discord.com/invite/PZqkgxu5SA) ### **BEE·bot** is a multipurpose scanner inspired by [Spiderfoot](https://github.com/smicallef/spiderfoot), built to automate your **Recon**, **Bug Bounties**, and **ASM**! @@ -20,7 +20,7 @@ pipx install --pip-args '\--pre' bbot _For more installation methods, including [Docker](https://hub.docker.com/r/blacklanternsecurity/bbot), see [Getting Started](https://www.blacklanternsecurity.com/bbot/Stable/)_ -> **Speed tip:** BBOT's DNS engine spins up ten workers per resolver in `/etc/resolv.conf`. Adding more unfiltered resolvers dramatically speeds up scans. See the [sample resolv.conf](docs/data/resolv-sample.conf) and [Tips and Tricks](https://www.blacklanternsecurity.com/bbot/Stable/scanning/tips_and_tricks/#speed-up-scans-with-more-dns-resolvers) for details. +> **Speed tip:** BBOT's DNS resolver ([blastdns](https://github.com/blacklanternsecurity/blastdns)) spins up multiple threads per resolver in `/etc/resolv.conf`. Adding more unfiltered resolvers dramatically speeds up scans. See the [sample resolv.conf](docs/data/resolv-sample.conf) and [Tips and Tricks](https://www.blacklanternsecurity.com/bbot/Stable/scanning/tips_and_tricks/#speed-up-scans-with-more-dns-resolvers) for details. ## Example Commands @@ -225,10 +225,18 @@ include: config: modules: + baddns: + enable_references: True dnsbrute: recursive_mutations: true dnscommonsrv: recursive_mutations: true + webbrute: + avoid_wafs: False + wayback: + urls: True + parameters: True + archive: True ``` @@ -386,6 +394,7 @@ For details, see [Configuration](https://www.blacklanternsecurity.com/bbot/Stabl - **Modules** - [List of Modules](https://www.blacklanternsecurity.com/bbot/Stable/modules/list_of_modules) - [Nuclei](https://www.blacklanternsecurity.com/bbot/Stable/modules/nuclei) + - [Wayback](https://www.blacklanternsecurity.com/bbot/Stable/modules/wayback) - [Custom YARA Rules](https://www.blacklanternsecurity.com/bbot/Stable/modules/custom_yara_rules) - [Lightfuzz (DAST)](https://www.blacklanternsecurity.com/bbot/Stable/modules/lightfuzz) - **Misc** @@ -395,6 +404,7 @@ For details, see [Configuration](https://www.blacklanternsecurity.com/bbot/Stabl - **Developer Manual** - [Development Overview](https://www.blacklanternsecurity.com/bbot/Stable/dev/) - [Setting Up a Dev Environment](https://www.blacklanternsecurity.com/bbot/Stable/dev/dev_environment) + - [Core Dependencies](https://www.blacklanternsecurity.com/bbot/Stable/dev/core_dependencies) - [BBOT Internal Architecture](https://www.blacklanternsecurity.com/bbot/Stable/dev/architecture) - [How to Write a BBOT Module](https://www.blacklanternsecurity.com/bbot/Stable/dev/module_howto) - [Validating & Inspecting Presets](https://www.blacklanternsecurity.com/bbot/Stable/dev/preset_validation) @@ -407,7 +417,6 @@ For details, see [Configuration](https://www.blacklanternsecurity.com/bbot/Stabl - [Target](https://www.blacklanternsecurity.com/bbot/Stable/dev/target) - [BaseModule](https://www.blacklanternsecurity.com/bbot/Stable/dev/basemodule) - [BBOTCore](https://www.blacklanternsecurity.com/bbot/Stable/dev/core) - - [Engine](https://www.blacklanternsecurity.com/bbot/Stable/dev/engine) - **Helpers** - [Overview](https://www.blacklanternsecurity.com/bbot/Stable/dev/helpers/) - [Command](https://www.blacklanternsecurity.com/bbot/Stable/dev/helpers/command) From 215b4a17a1ccce2dc47200764ec075d1cb3d67f6 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Wed, 17 Jun 2026 18:24:40 -0400 Subject: [PATCH 18/29] Fix DNS resolver thread counts in tips and tricks --- docs/scanning/tips_and_tricks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/scanning/tips_and_tricks.md b/docs/scanning/tips_and_tricks.md index f64c0972a7..81f34f0ad4 100644 --- a/docs/scanning/tips_and_tricks.md +++ b/docs/scanning/tips_and_tricks.md @@ -47,7 +47,7 @@ bbot -t evilcorp.com -f subdomain-enum -c dns.brute_threads=5000 ### Speed Up Scans with More DNS Resolvers -By far the most effective way to speed up a BBOT scan is to **add more resolvers to `/etc/resolv.conf`**. BBOT's DNS engine (blastdns) spins up ten workers per resolver, so more resolvers = more parallelism = faster scans. +By far the most effective way to speed up a BBOT scan is to **add more resolvers to `/etc/resolv.conf`**. BBOT's DNS resolver (blastdns) spins up multiple threads per resolver (default: `5`, configurable via `dns.threads`), so more resolvers = more parallelism = faster scans. For OSINT, it's critical that every resolver is **unfiltered**. Specialized resolvers that try to block ads, malicious domains, etc. will intentionally omit results. Below is a sample `/etc/resolv.conf` with 11 unfiltered public resolvers: @@ -55,7 +55,7 @@ For OSINT, it's critical that every resolver is **unfiltered**. Specialized reso --8<-- "docs/data/resolv-sample.conf" ``` -Copy this to `/etc/resolv.conf` (or append the `nameserver` lines to your existing config). With all 11 resolvers, blastdns will run 110 workers in parallel instead of the typical 10-30 you get from a default OS config. +Copy this to `/etc/resolv.conf` (or append the `nameserver` lines to your existing config). With all 11 resolvers at the default 5 threads each, blastdns will run 55 workers in parallel instead of the typical 5-15 you get from a default OS config. !!! tip If your system uses `systemd-resolved` or `resolvconf`, you may need to configure the upstream forwarders there instead of editing `/etc/resolv.conf` directly. From 8b1de5ed9ffa38a745a98018f4f415e4a4c33b07 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Fri, 19 Jun 2026 08:19:14 -0400 Subject: [PATCH 19/29] Migration guide: add JSON before/after, output module rules, omit_event_types --- docs/migration/3.0_breaking_changes.md | 269 +++++++++++++++++++++---- 1 file changed, 230 insertions(+), 39 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index 553a35e093..94060906f8 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -155,9 +155,9 @@ truth. The module called `http` in 3.0 is **not** the same as the output module called `http` in 2.x. The name was reassigned: - - **`http` (3.0, scan module)** — the replacement for the old `httpx` + - **`http` (3.0, scan module)** -- the replacement for the old `httpx` scan module. Probes URLs over the shared in-process blasthttp client. - - **`output.http` (2.x, output module)** — the webhook-style output module + - **`output.http` (2.x, output module)** -- the webhook-style output module that POSTed events to an arbitrary HTTP endpoint. This is now `output.webhook` in 3.0. @@ -167,8 +167,8 @@ truth. ### New modules worth knowing about -- `http` — replaces `httpx`; runs through the in-process [blasthttp](https://github.com/blacklanternsecurity/blasthttp) client. -- `webbrute` / `webbrute_shortnames` — ffuf replacements, also via blasthttp. +- `http` -- replaces `httpx`; runs through the in-process [blasthttp](https://github.com/blacklanternsecurity/blasthttp) client. +- `webbrute` / `webbrute_shortnames` -- ffuf replacements, also via blasthttp. - `bucket_hetzner`, `shodan_enterprise`, `trajan`, `legba`. - Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq`. - Lightfuzz submodules `esi` and `ssrf`. @@ -178,7 +178,7 @@ truth. - `BaseModule` no longer has `_event_handler_watchdog_task` as a class attribute and the watchdog is owned by `_setup()`. - New `BaseModule.update_event(event, **kwargs)` and `emit_event(existing_event, - ...)` flow. Passing an existing event to `make_event()` now raises — call + ...)` flow. Passing an existing event to `make_event()` now raises -- call `update_event()` instead. - New `BaseModule.setup_deps()` lifecycle hook (runs alongside `setup()` for pure dependency installation like AI models or wordlists). @@ -187,7 +187,7 @@ truth. - New `accept_seeds` attribute. Defaults to `True` for passive modules, `False` otherwise. Override explicitly if you want different behavior. - `default_discovery_context` now uses `{event.pretty_string}` instead of - `{event.data}` — the latter is now a dict for URL-like events (see below). + `{event.data}` -- the latter is now a dict for URL-like events (see below). - New `BaseModule._is_http_wildcard_host(event)` helper. Returns `True` when the target responds identically to two random paths (catch-all / SPA router). Used by `webbrute`, `lightfuzz`, and `paramminer` to skip hosts @@ -207,8 +207,8 @@ truth. | 2.x | 3.0 | |-----|-----| -| `URL_UNVERIFIED(BaseEvent)` — data is a string | `URL_UNVERIFIED(DictHostEvent)` — data is a dict with `url`, `path`, etc. | -| `URL(URL_UNVERIFIED)` — string data | `URL(URL_UNVERIFIED)` — dict data | +| `URL_UNVERIFIED(BaseEvent)` -- data is a string | `URL_UNVERIFIED(DictHostEvent)` -- data is a dict with `url`, `path`, etc. | +| `URL(URL_UNVERIFIED)` -- string data | `URL(URL_UNVERIFIED)` -- dict data | | `STORAGE_BUCKET(DictEvent, URL_UNVERIFIED)` | `STORAGE_BUCKET(URL_UNVERIFIED)` | | `HTTP_RESPONSE(URL_UNVERIFIED, DictEvent)` | `HTTP_RESPONSE(URL_UNVERIFIED)` | | `DictPathEvent(DictEvent)` | `DictPathEvent(DictHostEvent)` | @@ -235,11 +235,11 @@ that compared or formatted `event.data` for URL events must be updated. ### New event surface -- `event.url` — string URL property (works on URL-like events, returns `""` +- `event.url` -- string URL property (works on URL-like events, returns `""` otherwise). -- `event.pretty_string` — human-readable representation, used in logs and +- `event.pretty_string` -- human-readable representation, used in logs and discovery context. -- `event.host_metadata` — dict of structured per-host metadata (cloud +- `event.host_metadata` -- dict of structured per-host metadata (cloud providers, ASN info, etc.). Replaces the long-tail of `cloud-*` tags. - Mutation helpers `add_resolved_host()`, `update_resolved_hosts()`, `add_dns_child()`, `set_raw_dns_record()`. Direct assignment to the backing @@ -262,10 +262,143 @@ longer added directly. ### ASN as a target type -`ASN:12345` (or `AS12345`) can now be used as a scan target — the seed will be -expanded to its registered CIDRs via the `asndb` library at scan start. ASN -lookups are also wired through the new `bbot_io_api_key` config (or -`BBOT_IO_API_KEY` env var). +`ASN:12345` (or `AS12345`) can now be used as a scan target -- the seed will be +expanded to its registered CIDRs via the `asndb` library at scan start. If the +lookup fails (network error, API unavailable), BBOT retries 3 times with a +3-second delay. If all retries fail, the scan aborts with a message suggesting +you pass CIDR ranges directly instead. + +ASN lookups use the BLS API at `api.bbot.io`. No API key is required -- +unauthenticated users can operate at a reasonable pace. If you want higher rate +limits, set the `bbot_io_api_key` config value (or `BBOT_IO_API_KEY` env var). +Currently the ASN API is the only `api.bbot.io` service BBOT uses, but any +future BLS APIs will work with the same key. + +### SCAN event + +The `SCAN` event now includes a `network` key containing the scanner's hostname, +primary outbound IP, network interfaces (IPv4/IPv6 with netmasks), and default +routes. Useful for correlating scan activity back to a specific agent. + +### JSON output: before and after + +If you have downstream tooling that parses BBOT's NDJSON output, the schema +has changed in several ways. Here is a representative event from each version: + +**2.x -- URL event (string data)** + +```json +{ + "type": "URL", + "id": "URL:ab12cd34...", + "data": "https://www.evilcorp.com/login", + "host": "www.evilcorp.com", + "port": 443, + "resolved_hosts": ["1.2.3.4"], + "dns_children": {}, + "scope_distance": 0, + "scan": "SCAN:deadbeef...", + "timestamp": 1719792000.0, + "parent": "DNS_NAME:ef56ab78...", + "tags": ["in-scope", "status-200", "ip-1.2.3.4", "http-title-Login"], + "module": "httpx", + "module_sequence": "httpx" +} +``` + +**3.0 -- same URL event (dict data)** + +```json +{ + "type": "URL", + "id": "URL:ab12cd34...", + "uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479", + "data_json": { + "url": "https://www.evilcorp.com/login", + "status_code": 200, + "http_title": "Login" + }, + "host": "www.evilcorp.com", + "port": 443, + "resolved_hosts": ["1.2.3.4"], + "dns_children": {}, + "scope_distance": 0, + "scan": "SCAN:deadbeef...", + "timestamp": 1719792000.0, + "parent": "DNS_NAME:ef56ab78...", + "parent_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "tags": ["in-scope", "status-200"], + "module": "http", + "module_sequence": "http", + "discovery_context": "http probed www.evilcorp.com and found URL: https://www.evilcorp.com/login", + "discovery_path": ["SCAN:deadbeef", "DNS_NAME:www.evilcorp.com", "URL:https://www.evilcorp.com/login"], + "parent_chain": ["SCAN:deadbeef", "DNS_NAME:www.evilcorp.com"], + "host_metadata": {"cloud_provider": "amazon", "asn": 16509} +} +``` + +Key differences for parsers: + +| What changed | 2.x | 3.0 | +|---|---|---| +| URL event data | `"data": "https://..."` (string) | `"data_json": {"url": "https://...", ...}` (dict) | +| Non-URL event data | `"data": "..."` (string) | `"data": "..."` (still a string) | +| Module name for HTTP probing | `"module": "httpx"` | `"module": "http"` | +| Per-host cloud/IP tags | `"ip-1.2.3.4"`, `"http-title-Login"`, `"cloud-amazon"` in tags | Moved to `host_metadata` dict and `data_json` fields; tags simplified | +| Event ID | `"id"` only | `"id"` + new `"uuid"` (globally unique) | +| Parent reference | `"parent"` only | `"parent"` + new `"parent_uuid"` | +| Discovery chain | not present | `"discovery_context"`, `"discovery_path"`, `"parent_chain"` | + +!!! warning "The `data` vs `data_json` split" + When an event's `.data` is a **string** (DNS_NAME, IP_ADDRESS, EMAIL_ADDRESS, + OPEN_TCP_PORT, etc.), the JSON key is `"data"`. When `.data` is a **dict** + (URL, URL_UNVERIFIED, HTTP_RESPONSE, FINDING, STORAGE_BUCKET, etc.), the + JSON key is `"data_json"`. In 2.x, URL events used `"data"` with a string + value. In 3.0, they use `"data_json"` with a dict. Parsers that key on + `"data"` for URL events will silently get nothing back. + +**2.x -- VULNERABILITY event** + +```json +{ + "type": "VULNERABILITY", + "data": { + "host": "www.evilcorp.com", + "severity": "HIGH", + "description": "SQL injection in login form", + "url": "https://www.evilcorp.com/login" + }, + "tags": ["in-scope", "high"] +} +``` + +**3.0 -- equivalent FINDING event** + +```json +{ + "type": "FINDING", + "data_json": { + "host": "www.evilcorp.com", + "severity": "HIGH", + "confidence": "HIGH", + "name": "SQL Injection", + "description": "SQL injection in login form", + "url": "https://www.evilcorp.com/login" + }, + "tags": ["in-scope", "severity-high", "confidence-high"] +} +``` + +Changes: + +- `VULNERABILITY` type is gone; use `FINDING` instead. +- `confidence` is now **required** on every FINDING (one of `UNKNOWN`, `LOW`, + `MEDIUM`, `HIGH`, `CONFIRMED`). +- `name` is now **required** (short label for the finding). +- Severity strings changed: `INFORMATIONAL` is now `INFO`, `MODERATE` is now + `MEDIUM`. +- Tags changed from bare lowercase (`high`) to prefixed + (`severity-high`, `confidence-high`). --- @@ -279,7 +412,7 @@ lookups are also wired through the new `bbot_io_api_key` config (or |---------------------------|---------------------------| | `web.httpx_timeout` | `web.blasthttp_timeout` | | `web.httpx_retries` | `web.blasthttp_retries` | -| `dns.threads` (global) | `dns.threads` (now per-resolver; default lowered from 25 → 10) | +| `dns.threads` (global) | `dns.threads` (now per-resolver; default lowered from 25 -> 10) | | `web.ssl_verify` | split into `web.ssl_verify_target` (default `false`) and `web.ssl_verify_infrastructure` (default `true`) | ### Removed @@ -290,18 +423,19 @@ lookups are also wired through the new `bbot_io_api_key` config (or ### Added -- `max_mem_percent` — global ingress throttle when RSS exceeds the threshold. -- `web.user_agent_suffix` — appended to the user agent (previously buried as a +- `max_mem_percent` -- global ingress throttle when RSS exceeds the threshold. +- `web.user_agent_suffix` -- appended to the user agent (previously buried as a hidden CLI flag). -- `web.http_rate_limit` — global rps cap across the shared blasthttp client. -- `web.http_proxy_exclude` — hosts/CIDRs to exclude from the HTTP proxy +- `web.http_rate_limit` -- global rps cap across the shared blasthttp client. +- `web.http_proxy_exclude` -- hosts/CIDRs to exclude from the HTTP proxy (`NO_PROXY` equivalent). -- `web.body_spill.{enabled,cache_mb,compress}` — disk-spill HTTP response +- `web.body_spill.{enabled,cache_mb,compress}` -- disk-spill HTTP response bodies to keep them off the Python heap. -- `dns.cache_size` — DNS LRU size. -- `bbot_io_api_key` — API key for bbot.io services (currently used by ASN - lookups). -- `dns.abort_threshold` default lowered from `50` → `10`. +- `dns.cache_size` -- DNS LRU size. +- `bbot_io_api_key` -- optional API key for `api.bbot.io` services. Currently + only used by ASN lookups; no key is required for basic use. Future BLS APIs + will share this key. +- `dns.abort_threshold` default lowered from `50` -> `10`. - `dns.filter_ptrs` semantics: PTR-derived hostnames are now treated as affiliates by default rather than being injected as in-scope DNS_NAME events during IP-range scans. @@ -342,6 +476,55 @@ custom curl invocations must switch to `self.helpers.request(...)` / `zeromq`. - Neo4j output now also serializes `host_metadata`. +### Output modules are now purely additive + +In 2.x, specifying `-om` could silently replace the default output modules +depending on whether any of the names overlapped with the defaults. This was +confusing and inconsistent. + +In 3.0, `-om` is always **additive**: it adds modules on top of the defaults +(`csv`, `txt`, `json`, plus `stdout` from the CLI). To remove a default, use +the new `-eom` / `--exclude-output-modules` flag: + +```bash +# defaults (csv, txt, json, stdout) are all enabled +bbot -t evilcorp.com -p subdomain-enum + +# neo4j is added alongside all defaults +bbot -t evilcorp.com -p subdomain-enum -om neo4j + +# json only -- explicitly exclude the other defaults +bbot -t evilcorp.com -p subdomain-enum -eom csv txt stdout +``` + +The same applies in preset YAML (`exclude_output_modules:`) and the Python API +(`Preset(exclude_output_modules=[...])`). + +The `python` output module was also moved from `output/` to `internal/` -- it +was never a real output module, it is the Python API event bridge. + +### Omitted event types + +Certain event types are excluded from output by default via the +`omit_event_types` config. These events are still processed by modules +internally, but they do not appear in JSON, CSV, or stdout output unless +you remove them from the list: + +```yaml +omit_event_types: + - HTTP_RESPONSE + - RAW_TEXT + - URL_UNVERIFIED + - DNS_NAME_UNRESOLVED + - FILESYSTEM + - WEB_PARAMETER + - RAW_DNS_RECORD +``` + +If your downstream tooling relied on seeing `HTTP_RESPONSE` or +`URL_UNVERIFIED` events in the output, you will need to remove them from +`omit_event_types` in your preset or config. + --- ## Dependencies and tooling @@ -356,7 +539,7 @@ custom curl invocations must switch to `self.helpers.request(...)` / `<3.15`. - **Lockstep deps**: - `radixtarget >=4.0.1,<5` (composition pattern, no longer subclassed) - - `cloudcheck >=10.0.0,<11` + - `cloudcheck >=11.0.0,<12` - `blasthttp >=0.9.0` (new) - `blastdns >=1.9.0,<2` (new) - `asndb >=1.0.4` (new) @@ -375,27 +558,35 @@ subclasses, or `bbot.db.sql.models` will need to be ported. in scope. - [ ] Replace `-s` with `-S` for silent runs. - [ ] Drop `--allow-deadly`. -- [ ] Rename module references: `httpx → http`, `ffuf → webbrute`, - `bucket_azure → bucket_microsoft`, `extractous → kreuzberg`, - `output.http → output.webhook`, `censys → censys_dns / censys_ip`. -- [ ] Rename preset references: `web-basic → web`, `web-thorough → web-heavy`, - `*-intense → *-heavy`. -- [ ] Rename module flags: `noisy → loud`, `web-basic → web`, - `web-thorough → web-heavy`. Drop `aggressive` / `deadly`. Add `safe` / +- [ ] Rename module references: `httpx -> http`, `ffuf -> webbrute`, + `bucket_azure -> bucket_microsoft`, `extractous -> kreuzberg`, + `output.http -> output.webhook`, `censys -> censys_dns / censys_ip`. +- [ ] Rename preset references: `web-basic -> web`, `web-thorough -> web-heavy`, + `*-intense -> *-heavy`. +- [ ] Rename module flags: `noisy -> loud`, `web-basic -> web`, + `web-thorough -> web-heavy`. Drop `aggressive` / `deadly`. Add `safe` / `loud` / `invasive` to satisfy the new validation rule. - [ ] Replace any `VULNERABILITY` emit with a `FINDING` carrying `severity=...`. -- [ ] Update FINDING severity strings: `INFORMATIONAL → INFO`, - `MODERATE → MEDIUM`. Add a `confidence` field from the new allowlist. +- [ ] Update FINDING severity strings: `INFORMATIONAL -> INFO`, + `MODERATE -> MEDIUM`. Add a `confidence` field from the new allowlist. - [ ] Stop reading `event.data` for URL events; use `event.url` / `event.pretty_string`. Drop dependencies on `event.confidence` / `cumulative_confidence`. - [ ] Stop mutating `event._resolved_hosts` / `event.dns_children` directly; use the new `add_resolved_host` / `add_dns_child` helpers. -- [ ] Update config keys: `web.httpx_timeout → web.blasthttp_timeout`, - `web.httpx_retries → web.blasthttp_retries`, - `web.ssl_verify → web.ssl_verify_target` / `web.ssl_verify_infrastructure`. -- [ ] Update imports: `bbot.db.sql.models → bbot.models.sql`. +- [ ] Update config keys: `web.httpx_timeout -> web.blasthttp_timeout`, + `web.httpx_retries -> web.blasthttp_retries`, + `web.ssl_verify -> web.ssl_verify_target` / `web.ssl_verify_infrastructure`. +- [ ] Update imports: `bbot.db.sql.models -> bbot.models.sql`. - [ ] If you ship a custom module that watches URLs but doesn't want blasthttp auto-enabled, set `_disable_auto_module_deps = True`. - [ ] Switch your install/build pipeline from Poetry to uv. +- [ ] If you relied on `-om` to replace default output modules, switch to + `-eom` to exclude the ones you don't want (output modules are now + purely additive). +- [ ] Update any JSON parsers that read `event["data"]` for URL events -- + the key is now `"data_json"` and the value is a dict, not a string. + Use `event["data_json"]["url"]` to get the URL string. +- [ ] If your tooling consumed `HTTP_RESPONSE` or `URL_UNVERIFIED` from + output, remove them from `omit_event_types` in your config. From df81eb3c3d6393980ab1e501038c6ec082e11578 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Fri, 19 Jun 2026 14:22:21 -0400 Subject: [PATCH 20/29] Update docs for timeout consolidation and ssl_verify split --- docs/scanning/configuration.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/scanning/configuration.md b/docs/scanning/configuration.md index ef2f680956..6bdacb6db3 100644 --- a/docs/scanning/configuration.md +++ b/docs/scanning/configuration.md @@ -164,10 +164,10 @@ web: spider_depth: 1 # Set the maximum number of links that can be followed per page spider_links_per_page: 25 - # HTTP timeout (for Python requests; API calls, etc.) + # HTTP timeout for target-directed traffic (probes, crawls, etc.) http_timeout: 10 - # HTTP timeout (for blasthttp) - blasthttp_timeout: 5 + # HTTP timeout for non-target traffic (APIs, wordlist downloads, etc.) + http_timeout_infrastructure: 10 # Custom HTTP headers (e.g. cookies, etc.) # in the format { "Header-Key": "header_value" } # These are attached to all in-scope HTTP requests @@ -179,8 +179,6 @@ web: api_retries: 2 # HTTP retries - try again if the raw connection fails http_retries: 1 - # HTTP retries (for blasthttp) - blasthttp_retries: 1 # Default sleep interval when rate limited by 429 (and retry-after isn't provided) 429_sleep_interval: 30 # Maximum sleep interval when rate limited by 429 (and an excessive retry-after is provided) @@ -189,8 +187,10 @@ web: debug: false # Maximum number of HTTP redirects to follow http_max_redirects: 5 - # Whether to verify SSL certificates - ssl_verify: false + # Whether to verify SSL certificates for target-directed traffic (probes, crawls, etc.) + ssl_verify_target: false + # Whether to verify SSL certificates for non-target traffic (APIs, wordlist downloads, etc.) + ssl_verify_infrastructure: true # Maximum HTTP requests per second (0 = unlimited) # Applies globally across all blasthttp consumers (http probing, web brute, etc.) http_rate_limit: 0 From f3575cc5fbdc1c085663918b96b9a2249484995a Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Fri, 19 Jun 2026 14:25:11 -0400 Subject: [PATCH 21/29] Update migration guide for timeout consolidation and ssl_verify split --- docs/migration/3.0_breaking_changes.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index 94060906f8..efe0dfd207 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -410,10 +410,12 @@ Changes: | 2.x | 3.0 | |---------------------------|---------------------------| -| `web.httpx_timeout` | `web.blasthttp_timeout` | -| `web.httpx_retries` | `web.blasthttp_retries` | +| `web.httpx_timeout` | `web.http_timeout` (target-directed traffic, default `10`) | +| *(no equivalent)* | `web.http_timeout_infrastructure` (API calls, wordlist downloads, etc., default `10`) | +| `web.httpx_retries` | `web.http_retries` | | `dns.threads` (global) | `dns.threads` (now per-resolver; default lowered from 25 -> 10) | -| `web.ssl_verify` | split into `web.ssl_verify_target` (default `false`) and `web.ssl_verify_infrastructure` (default `true`) | +| `web.ssl_verify` | `web.ssl_verify_target` (target-directed traffic, default `false`) | +| *(no equivalent)* | `web.ssl_verify_infrastructure` (API calls, wordlist downloads, etc., default `true`) | ### Removed @@ -575,8 +577,9 @@ subclasses, or `bbot.db.sql.models` will need to be ported. `cumulative_confidence`. - [ ] Stop mutating `event._resolved_hosts` / `event.dns_children` directly; use the new `add_resolved_host` / `add_dns_child` helpers. -- [ ] Update config keys: `web.httpx_timeout -> web.blasthttp_timeout`, - `web.httpx_retries -> web.blasthttp_retries`, +- [ ] Update config keys: `web.httpx_timeout -> web.http_timeout` / + `web.http_timeout_infrastructure`, + `web.httpx_retries -> web.http_retries`, `web.ssl_verify -> web.ssl_verify_target` / `web.ssl_verify_infrastructure`. - [ ] Update imports: `bbot.db.sql.models -> bbot.models.sql`. - [ ] If you ship a custom module that watches URLs but doesn't want From cdc27711dcc2296ea228d425d7a2d6f9cd3cdd03 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Thu, 25 Jun 2026 22:50:10 -0400 Subject: [PATCH 22/29] Migration guide: stale-config reset flags, vhost rename, wpscan removal --- docs/migration/3.0_breaking_changes.md | 64 +++++++++++++++++++++++++- 1 file changed, 62 insertions(+), 2 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index efe0dfd207..528b6d8ab4 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -132,8 +132,9 @@ gone too; inject secrets with shell expansion - `passivetotal` - `sitedossier` - `smuggler` -- `vhost` - `wappalyzer` +- `wpscan` (dropped over a heavy Ruby/compiler install footprint and an + unmaintained upstream) ### Removed and replaced @@ -145,6 +146,7 @@ gone too; inject secrets with shell expansion | `bucket_azure` | `bucket_microsoft` | | `extractous` | `kreuzberg` | | `output.http` | `output.webhook` | +| `vhost` | `virtualhost` (rebuilt; emits the new `VIRTUAL_HOST` event) | | `censys` (single module) | split into `censys_dns` and `censys_ip` | If you used any of these in a custom preset or `-m` / `-em` invocation, update @@ -202,6 +204,8 @@ truth. - `VULNERABILITY` is gone. Emit a `FINDING` with `severity` set to `"CRITICAL"`, `"HIGH"`, `"MEDIUM"`, `"LOW"`, or `"INFO"` instead. The pseudo event type `TARGET` was also retired in favor of `SEED`. +- `VHOST` was renamed to `VIRTUAL_HOST`, emitted by the rebuilt `virtualhost` + module (formerly `vhost`). ### Class hierarchy @@ -274,6 +278,13 @@ limits, set the `bbot_io_api_key` config value (or `BBOT_IO_API_KEY` env var). Currently the ASN API is the only `api.bbot.io` service BBOT uses, but any future BLS APIs will work with the same key. +ASN *enrichment* (the ASN/subnet data attached to in-scope IPs via +`host_metadata`) degrades gracefully rather than aborting: after several +consecutive lookup failures, a circuit breaker disables ASN enrichment for the +rest of the scan and the scan continues without it. This is separate from the +ASN-as-target expansion above, which must abort because there would be nothing +to scan. + ### SCAN event The `SCAN` event now includes a `network` key containing the scanner's hostname, @@ -442,6 +453,49 @@ Changes: affiliates by default rather than being injected as in-scope DNS_NAME events during IP-range scans. +### Stale config files and `--reset-config` / `--reset-secrets` + +BBOT generates two config files the first time it runs and then never +overwrites them: + +- `~/.config/bbot/bbot.yml` -- a fully-commented snapshot of all default options +- `~/.config/bbot/secrets.yml` -- the secret-bearing subset (API keys, etc.), + written owner-only (`0600`) + +Because they're written once and left alone, they drift out of sync with the +defaults as you upgrade. A `bbot.yml` generated under 2.x can still mention +`web.httpx_timeout`, `web.ssl_verify`, `modules.json.siem_friendly`, and other +keys that were renamed or removed in 3.0. + +In 2.x this was harmless: unknown keys were silently ignored. In 3.0, config is +**validated on load**, so a leftover key in your generated file now produces a +validation error before the scan starts. When the offending key lives in one of +these generated files (rather than being a `-c` typo on the command line), BBOT +tells you which file it came from and points you at the matching reset flag. + +Two new CLI flags regenerate the files from current defaults: + +```bash +bbot --reset-config # regenerate bbot.yml +bbot --reset-secrets # regenerate secrets.yml +``` + +- **Destructive.** A regenerated file is a fresh, fully-commented template, so + any options you had *uncommented* are wiped. The existing file is backed up + first to `<name>.bak` (then `.bak.1`, `.bak.2`, ... so earlier backups aren't + clobbered). +- **Confirmation required.** You're prompted before anything is overwritten; + pass `-y` / `--yes` to skip the prompt. Non-interactive runs without `--yes` + refuse and exit rather than overwrite silently. +- **Independent.** The two files are reset separately: `--reset-config` never + touches the API keys in `secrets.yml`, and `--reset-secrets` never touches the + tuned options in `bbot.yml`. The regenerated `secrets.yml` is written + atomically and stays owner-only. + +The usual fix after an in-place upgrade is to copy any customizations out of the +old file, run the matching reset flag, then re-apply your changes to the fresh +template (or move them into a preset). + --- ## DNS and HTTP internals @@ -562,7 +616,9 @@ subclasses, or `bbot.db.sql.models` will need to be ported. - [ ] Drop `--allow-deadly`. - [ ] Rename module references: `httpx -> http`, `ffuf -> webbrute`, `bucket_azure -> bucket_microsoft`, `extractous -> kreuzberg`, - `output.http -> output.webhook`, `censys -> censys_dns / censys_ip`. + `output.http -> output.webhook`, `vhost -> virtualhost`, + `censys -> censys_dns / censys_ip`. Drop any `wpscan` references (removed, + no replacement). - [ ] Rename preset references: `web-basic -> web`, `web-thorough -> web-heavy`, `*-intense -> *-heavy`. - [ ] Rename module flags: `noisy -> loud`, `web-basic -> web`, @@ -581,6 +637,10 @@ subclasses, or `bbot.db.sql.models` will need to be ported. `web.http_timeout_infrastructure`, `web.httpx_retries -> web.http_retries`, `web.ssl_verify -> web.ssl_verify_target` / `web.ssl_verify_infrastructure`. +- [ ] If an in-place upgrade fails with a config validation error pointing at + `~/.config/bbot/bbot.yml` or `secrets.yml`, back up your customizations + and run `bbot --reset-config` / `bbot --reset-secrets` to regenerate from + current defaults. - [ ] Update imports: `bbot.db.sql.models -> bbot.models.sql`. - [ ] If you ship a custom module that watches URLs but doesn't want blasthttp auto-enabled, set `_disable_auto_module_deps = True`. From f26b6b188995eb3783039a2f076c4993b3567e4e Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 18:25:16 -0400 Subject: [PATCH 23/29] Regenerate docs from current module roster Run bbot/scripts/docs.py to refresh all generated tables (modules, presets, events, config, flags, output, help, TOC) and the chord-graph data. Sweeps removed modules (wpscan, httpx, ffuf, azure_realm, etc.) and adds new ones (http, webbrute, censys_dns/ip, mongo/kafka/...). --- README.md | 2 +- docs/data/chord_graph/entities.json | 564 ++++++++++++----------- docs/data/chord_graph/rels.json | 463 ++++++++++--------- docs/modules/list_of_modules.md | 9 +- docs/modules/nuclei.md | 32 +- docs/scanning/advanced.md | 56 +-- docs/scanning/configuration.md | 676 +++++++++++++++------------- docs/scanning/events.md | 70 +-- docs/scanning/index.md | 50 +- docs/scanning/output.md | 14 +- docs/scanning/presets_list.md | 214 +++++---- 11 files changed, 1149 insertions(+), 1001 deletions(-) diff --git a/README.md b/README.md index 2b4b69602c..3f08a21b6d 100644 --- a/README.md +++ b/README.md @@ -396,7 +396,7 @@ For details, see [Configuration](https://www.blacklanternsecurity.com/bbot/Stabl - [Nuclei](https://www.blacklanternsecurity.com/bbot/Stable/modules/nuclei) - [Wayback](https://www.blacklanternsecurity.com/bbot/Stable/modules/wayback) - [Custom YARA Rules](https://www.blacklanternsecurity.com/bbot/Stable/modules/custom_yara_rules) - - [Lightfuzz (DAST)](https://www.blacklanternsecurity.com/bbot/Stable/modules/lightfuzz) + - [Lightfuzz](https://www.blacklanternsecurity.com/bbot/Stable/modules/lightfuzz) - **Misc** - [Contribution](https://www.blacklanternsecurity.com/bbot/Stable/contribution) - [Release History](https://www.blacklanternsecurity.com/bbot/Stable/release_history) diff --git a/docs/data/chord_graph/entities.json b/docs/data/chord_graph/entities.json index c1be70a10c..1aa25f459c 100644 --- a/docs/data/chord_graph/entities.json +++ b/docs/data/chord_graph/entities.json @@ -38,23 +38,23 @@ "name": "CODE_REPOSITORY", "parent": 88888888, "consumes": [ - 64, - 79, + 65, 80, - 84, - 87, + 81, + 85, + 88, 123, 141, 143 ], "produces": [ 45, - 65, - 78, - 81, + 66, + 79, 82, - 85, + 83, 86, + 87, 122 ] }, @@ -91,20 +91,20 @@ 59, 60, 61, - 63, - 69, - 76, - 81, - 83, - 91, - 95, - 104, - 108, - 110, - 113, + 64, + 70, + 77, + 82, + 84, + 92, + 96, + 105, + 109, + 111, 114, - 118, - 120, + 115, + 119, + 121, 124, 128, 129, @@ -118,8 +118,8 @@ 142, 146, 147, - 148, - 150 + 151, + 154 ], "produces": [ 6, @@ -140,14 +140,14 @@ 60, 61, 62, - 76, - 91, - 95, - 104, - 108, - 111, - 113, + 77, + 92, + 96, + 105, + 109, + 112, 114, + 115, 124, 128, 130, @@ -159,8 +159,8 @@ 142, 146, 147, - 148, - 150 + 151, + 154 ] }, { @@ -174,22 +174,31 @@ ], "produces": [] }, + { + "id": 149, + "name": "DNS_NAME_UNVERIFIED", + "parent": 88888888, + "consumes": [], + "produces": [ + 148 + ] + }, { "id": 48, "name": "EMAIL_ADDRESS", "parent": 88888888, "consumes": [ - 70 + 71 ], "produces": [ 47, 54, 59, - 63, - 69, - 83, - 95, - 118, + 64, + 70, + 84, + 96, + 119, 129, 133, 136 @@ -200,19 +209,19 @@ "name": "FILESYSTEM", "parent": 88888888, "consumes": [ - 102, 103, + 104, 143, 144 ], "produces": [ 8, - 64, - 74, - 79, + 65, + 75, 80, - 84, - 102, + 81, + 85, + 103, 123, 144 ] @@ -223,7 +232,7 @@ "parent": 88888888, "consumes": [ 15, - 152 + 156 ], "produces": [ 1, @@ -240,19 +249,19 @@ 33, 34, 37, - 68, - 77, + 69, 78, - 86, - 90, - 92, - 94, - 105, + 79, + 87, + 91, + 93, + 95, 106, 107, - 109, - 111, + 108, + 110, 112, + 113, 125, 126, 131, @@ -262,17 +271,18 @@ 141, 143, 145, - 155 + 152, + 154 ] }, { - "id": 99, + "id": 100, "name": "GEOLOCATION", "parent": 88888888, "consumes": [], "produces": [ - 98, - 101 + 99, + 102 ] }, { @@ -293,25 +303,27 @@ 1, 15, 26, - 68, - 71, - 74, - 86, - 92, - 109, + 69, + 72, + 75, + 87, + 93, 110, 111, - 115, + 112, 116, 117, + 118, 135, 136, 140, - 143, - 155 + 143 ], "produces": [ - 93 + 94, + 107, + 148, + 154 ] }, { @@ -322,11 +334,11 @@ 11, 15, 40, - 98, - 100, + 99, 101, - 110, - 120, + 102, + 111, + 121, 131, 132, 135 @@ -335,19 +347,21 @@ 15, 40, 62, - 100, + 101, 135 ] }, { - "id": 121, + "id": 63, "name": "IP_RANGE", "parent": 88888888, "consumes": [ - 120, + 121, 135 ], - "produces": [] + "produces": [ + 62 + ] }, { "id": 9, @@ -357,7 +371,7 @@ 8 ], "produces": [ - 87 + 88 ] }, { @@ -366,15 +380,15 @@ "parent": 88888888, "consumes": [ 15, - 75, - 93, - 110, - 119 + 76, + 94, + 111, + 120 ], "produces": [ 15, 40, - 120, + 121, 131, 132, 135 @@ -391,13 +405,13 @@ ] }, { - "id": 66, + "id": 67, "name": "ORG_STUB", "parent": 88888888, "consumes": [ - 65, - 82, - 87, + 66, + 83, + 88, 122 ], "produces": [ @@ -419,13 +433,13 @@ "name": "PROTOCOL", "parent": 88888888, "consumes": [ - 105, - 107, - 110 + 106, + 108, + 111 ], "produces": [ 40, - 75 + 76 ] }, { @@ -436,38 +450,38 @@ "produces": [ 55, 62, - 63 + 64 ] }, { - "id": 72, + "id": 73, "name": "RAW_TEXT", "parent": 88888888, "consumes": [ - 71, + 72, 143 ], "produces": [ - 103 + 104 ] }, { - "id": 67, + "id": 68, "name": "SOCIAL", "parent": 88888888, "consumes": [ - 65, - 82, - 85, + 66, + 83, 86, - 88, + 87, + 89, 122, 135 ], "produces": [ - 65, - 83, - 86, + 66, + 84, + 87, 134 ] }, @@ -501,22 +515,20 @@ "parent": 88888888, "consumes": [ 15, - 86, + 87, 141, - 152, - 155 + 156 ], "produces": [ 1, 26, 40, - 68, - 86, - 88, - 112, + 69, + 87, + 89, + 113, 131, - 132, - 155 + 132 ] }, { @@ -529,38 +541,41 @@ 15, 23, 37, - 77, 78, - 88, - 90, - 93, - 96, - 106, - 111, + 79, + 89, + 91, + 94, + 97, + 107, 112, - 119, + 113, + 120, 127, 135, 140, 145, - 149, + 148, 152, - 153 + 153, + 154, + 156, + 157 ], "produces": [ - 88, - 93 + 89, + 94 ] }, { - "id": 97, + "id": 98, "name": "URL_HINT", "parent": 88888888, "consumes": [ - 154 + 158 ], "produces": [ - 96 + 97 ] }, { @@ -569,10 +584,10 @@ "parent": 88888888, "consumes": [ 45, - 74, - 93, - 113, - 119, + 75, + 94, + 114, + 120, 126, 134, 135, @@ -585,20 +600,19 @@ 40, 55, 59, - 63, - 65, - 71, - 75, - 81, - 88, - 95, + 64, + 66, + 72, + 76, + 82, + 89, + 96, 127, 129, 146, - 150, - 153, 154, - 155 + 157, + 158 ] }, { @@ -613,6 +627,15 @@ 54 ] }, + { + "id": 150, + "name": "VIRTUAL_HOST", + "parent": 88888888, + "consumes": [], + "produces": [ + 148 + ] + }, { "id": 17, "name": "WAF", @@ -621,36 +644,37 @@ 15 ], "produces": [ - 149 + 153 ] }, { - "id": 89, + "id": 90, "name": "WEBSCREENSHOT", "parent": 88888888, "consumes": [], "produces": [ - 88 + 89 ] }, { - "id": 73, + "id": 74, "name": "WEB_PARAMETER", "parent": 88888888, "consumes": [ - 94, - 106, - 115, + 95, + 107, 116, 117, + 118, 125, - 151 + 155 ], "produces": [ - 71, - 115, + 72, 116, - 117 + 117, + 118, + 154 ] }, { @@ -1123,11 +1147,12 @@ "produces": [ 7, 12, + 63, 56 ] }, { - "id": 63, + "id": 64, "name": "dnstlsrpt", "parent": 99999999, "consumes": [ @@ -1140,7 +1165,7 @@ ] }, { - "id": 64, + "id": 65, "name": "docker_pull", "parent": 99999999, "consumes": [ @@ -1151,21 +1176,21 @@ ] }, { - "id": 65, + "id": 66, "name": "dockerhub", "parent": 99999999, "consumes": [ - 66, - 67 + 67, + 68 ], "produces": [ 46, - 67, + 68, 20 ] }, { - "id": 68, + "id": 69, "name": "dotnetnuke", "parent": 99999999, "consumes": [ @@ -1177,7 +1202,7 @@ ] }, { - "id": 69, + "id": 70, "name": "emailformat", "parent": 99999999, "consumes": [ @@ -1188,7 +1213,7 @@ ] }, { - "id": 70, + "id": 71, "name": "emails", "parent": 99999999, "consumes": [ @@ -1197,20 +1222,20 @@ "produces": [] }, { - "id": 71, + "id": 72, "name": "excavate", "parent": 99999999, "consumes": [ 2, - 72 + 73 ], "produces": [ 20, - 73 + 74 ] }, { - "id": 74, + "id": 75, "name": "filedownload", "parent": 99999999, "consumes": [ @@ -1222,7 +1247,7 @@ ] }, { - "id": 75, + "id": 76, "name": "fingerprintx", "parent": 99999999, "consumes": [ @@ -1234,7 +1259,7 @@ ] }, { - "id": 76, + "id": 77, "name": "fullhunt", "parent": 99999999, "consumes": [ @@ -1245,7 +1270,7 @@ ] }, { - "id": 77, + "id": 78, "name": "generic_ssrf", "parent": 99999999, "consumes": [ @@ -1256,7 +1281,7 @@ ] }, { - "id": 78, + "id": 79, "name": "git", "parent": 99999999, "consumes": [ @@ -1268,7 +1293,7 @@ ] }, { - "id": 79, + "id": 80, "name": "git_clone", "parent": 99999999, "consumes": [ @@ -1279,7 +1304,7 @@ ] }, { - "id": 80, + "id": 81, "name": "gitdumper", "parent": 99999999, "consumes": [ @@ -1290,7 +1315,7 @@ ] }, { - "id": 81, + "id": 82, "name": "github_codesearch", "parent": 99999999, "consumes": [ @@ -1302,19 +1327,19 @@ ] }, { - "id": 82, + "id": 83, "name": "github_org", "parent": 99999999, "consumes": [ - 66, - 67 + 67, + 68 ], "produces": [ 46 ] }, { - "id": 83, + "id": 84, "name": "github_usersearch", "parent": 99999999, "consumes": [ @@ -1322,11 +1347,11 @@ ], "produces": [ 48, - 67 + 68 ] }, { - "id": 84, + "id": 85, "name": "github_workflows", "parent": 99999999, "consumes": [ @@ -1337,61 +1362,61 @@ ] }, { - "id": 85, + "id": 86, "name": "gitlab_com", "parent": 99999999, "consumes": [ - 67 + 68 ], "produces": [ 46 ] }, { - "id": 86, + "id": 87, "name": "gitlab_onprem", "parent": 99999999, "consumes": [ 2, - 67, + 68, 5 ], "produces": [ 46, 4, - 67, + 68, 5 ] }, { - "id": 87, + "id": 88, "name": "google_playstore", "parent": 99999999, "consumes": [ 46, - 66 + 67 ], "produces": [ 9 ] }, { - "id": 88, + "id": 89, "name": "gowitness", "parent": 99999999, "consumes": [ - 67, + 68, 3 ], "produces": [ 5, 3, 20, - 89 + 90 ] }, { - "id": 90, + "id": 91, "name": "graphql_introspection", "parent": 99999999, "consumes": [ @@ -1402,7 +1427,7 @@ ] }, { - "id": 91, + "id": 92, "name": "hackertarget", "parent": 99999999, "consumes": [ @@ -1413,7 +1438,7 @@ ] }, { - "id": 92, + "id": 93, "name": "host_header", "parent": 99999999, "consumes": [ @@ -1424,7 +1449,7 @@ ] }, { - "id": 93, + "id": 94, "name": "http", "parent": 99999999, "consumes": [ @@ -1438,18 +1463,18 @@ ] }, { - "id": 94, + "id": 95, "name": "hunt", "parent": 99999999, "consumes": [ - 73 + 74 ], "produces": [ 4 ] }, { - "id": 95, + "id": 96, "name": "hunterio", "parent": 99999999, "consumes": [ @@ -1462,29 +1487,29 @@ ] }, { - "id": 96, + "id": 97, "name": "iis_shortnames", "parent": 99999999, "consumes": [ 3 ], "produces": [ - 97 + 98 ] }, { - "id": 98, + "id": 99, "name": "ip2location", "parent": 99999999, "consumes": [ 12 ], "produces": [ - 99 + 100 ] }, { - "id": 100, + "id": 101, "name": "ipneighbor", "parent": 99999999, "consumes": [ @@ -1495,18 +1520,18 @@ ] }, { - "id": 101, + "id": 102, "name": "ipstack", "parent": 99999999, "consumes": [ 12 ], "produces": [ - 99 + 100 ] }, { - "id": 102, + "id": 103, "name": "jadx", "parent": 99999999, "consumes": [ @@ -1517,18 +1542,18 @@ ] }, { - "id": 103, + "id": 104, "name": "kreuzberg", "parent": 99999999, "consumes": [ 10 ], "produces": [ - 72 + 73 ] }, { - "id": 104, + "id": 105, "name": "leakix", "parent": 99999999, "consumes": [ @@ -1539,7 +1564,7 @@ ] }, { - "id": 105, + "id": 106, "name": "legba", "parent": 99999999, "consumes": [ @@ -1550,19 +1575,20 @@ ] }, { - "id": 106, + "id": 107, "name": "lightfuzz", "parent": 99999999, "consumes": [ 3, - 73 + 74 ], "produces": [ - 4 + 4, + 2 ] }, { - "id": 107, + "id": 108, "name": "medusa", "parent": 99999999, "consumes": [ @@ -1573,7 +1599,7 @@ ] }, { - "id": 108, + "id": 109, "name": "myssl", "parent": 99999999, "consumes": [ @@ -1584,7 +1610,7 @@ ] }, { - "id": 109, + "id": 110, "name": "newsletters", "parent": 99999999, "consumes": [ @@ -1595,7 +1621,7 @@ ] }, { - "id": 110, + "id": 111, "name": "nmap_xml", "parent": 99999999, "consumes": [ @@ -1608,7 +1634,7 @@ "produces": [] }, { - "id": 111, + "id": 112, "name": "ntlm", "parent": 99999999, "consumes": [ @@ -1621,7 +1647,7 @@ ] }, { - "id": 112, + "id": 113, "name": "nuclei", "parent": 99999999, "consumes": [ @@ -1633,7 +1659,7 @@ ] }, { - "id": 113, + "id": 114, "name": "oauth", "parent": 99999999, "consumes": [ @@ -1645,7 +1671,7 @@ ] }, { - "id": 114, + "id": 115, "name": "otx", "parent": 99999999, "consumes": [ @@ -1656,43 +1682,43 @@ ] }, { - "id": 115, + "id": 116, "name": "paramminer_cookies", "parent": 99999999, "consumes": [ 2, - 73 + 74 ], "produces": [ - 73 + 74 ] }, { - "id": 116, + "id": 117, "name": "paramminer_getparams", "parent": 99999999, "consumes": [ 2, - 73 + 74 ], "produces": [ - 73 + 74 ] }, { - "id": 117, + "id": 118, "name": "paramminer_headers", "parent": 99999999, "consumes": [ 2, - 73 + 74 ], "produces": [ - 73 + 74 ] }, { - "id": 118, + "id": 119, "name": "pgp", "parent": 99999999, "consumes": [ @@ -1703,7 +1729,7 @@ ] }, { - "id": 119, + "id": 120, "name": "portfilter", "parent": 99999999, "consumes": [ @@ -1714,13 +1740,13 @@ "produces": [] }, { - "id": 120, + "id": 121, "name": "portscan", "parent": 99999999, "consumes": [ 7, 12, - 121 + 63 ], "produces": [ 16 @@ -1731,8 +1757,8 @@ "name": "postman", "parent": 99999999, "consumes": [ - 66, - 67 + 67, + 68 ], "produces": [ 46 @@ -1765,7 +1791,7 @@ "name": "reflected_parameters", "parent": 99999999, "consumes": [ - 73 + 74 ], "produces": [ 4 @@ -1875,7 +1901,7 @@ 20 ], "produces": [ - 67 + 68 ] }, { @@ -1888,8 +1914,8 @@ 22, 2, 12, - 121, - 67, + 63, + 68, 24, 3, 20, @@ -1900,7 +1926,7 @@ 4, 12, 16, - 66 + 67 ] }, { @@ -1991,7 +2017,7 @@ 46, 10, 2, - 72 + 73 ], "produces": [ 4 @@ -2044,6 +2070,19 @@ }, { "id": 148, + "name": "virtualhost", + "parent": 99999999, + "consumes": [ + 3 + ], + "produces": [ + 149, + 2, + 150 + ] + }, + { + "id": 151, "name": "virustotal", "parent": 99999999, "consumes": [ @@ -2054,7 +2093,18 @@ ] }, { - "id": 149, + "id": 152, + "name": "waf_bypass", + "parent": 99999999, + "consumes": [ + 3 + ], + "produces": [ + 4 + ] + }, + { + "id": 153, "name": "wafw00f", "parent": 99999999, "consumes": [ @@ -2065,28 +2115,32 @@ ] }, { - "id": 150, + "id": 154, "name": "wayback", "parent": 99999999, "consumes": [ - 7 + 7, + 3 ], "produces": [ 7, - 20 + 4, + 2, + 20, + 74 ] }, { - "id": 151, + "id": 155, "name": "web_parameters", "parent": 99999999, "consumes": [ - 73 + 74 ], "produces": [] }, { - "id": 152, + "id": 156, "name": "web_report", "parent": 99999999, "consumes": [ @@ -2097,7 +2151,7 @@ "produces": [] }, { - "id": 153, + "id": 157, "name": "webbrute", "parent": 99999999, "consumes": [ @@ -2108,27 +2162,13 @@ ] }, { - "id": 154, + "id": 158, "name": "webbrute_shortnames", "parent": 99999999, "consumes": [ - 97 - ], - "produces": [ - 20 - ] - }, - { - "id": 155, - "name": "wpscan", - "parent": 99999999, - "consumes": [ - 2, - 5 + 98 ], "produces": [ - 4, - 5, 20 ] } diff --git a/docs/data/chord_graph/rels.json b/docs/data/chord_graph/rels.json index e3cf6566c3..c816b6db22 100644 --- a/docs/data/chord_graph/rels.json +++ b/docs/data/chord_graph/rels.json @@ -609,739 +609,749 @@ "target": 62, "type": "produces" }, + { + "source": 63, + "target": 62, + "type": "produces" + }, { "source": 56, "target": 62, "type": "produces" }, { - "source": 63, + "source": 64, "target": 7, "type": "consumes" }, { "source": 48, - "target": 63, + "target": 64, "type": "produces" }, { "source": 56, - "target": 63, + "target": 64, "type": "produces" }, { "source": 20, - "target": 63, + "target": 64, "type": "produces" }, { - "source": 64, + "source": 65, "target": 46, "type": "consumes" }, { "source": 10, - "target": 64, + "target": 65, "type": "produces" }, { - "source": 65, - "target": 66, + "source": 66, + "target": 67, "type": "consumes" }, { - "source": 65, - "target": 67, + "source": 66, + "target": 68, "type": "consumes" }, { "source": 46, - "target": 65, + "target": 66, "type": "produces" }, { - "source": 67, - "target": 65, + "source": 68, + "target": 66, "type": "produces" }, { "source": 20, - "target": 65, + "target": 66, "type": "produces" }, { - "source": 68, + "source": 69, "target": 2, "type": "consumes" }, { "source": 4, - "target": 68, + "target": 69, "type": "produces" }, { "source": 5, - "target": 68, + "target": 69, "type": "produces" }, { - "source": 69, + "source": 70, "target": 7, "type": "consumes" }, { "source": 48, - "target": 69, + "target": 70, "type": "produces" }, { - "source": 70, + "source": 71, "target": 48, "type": "consumes" }, { - "source": 71, + "source": 72, "target": 2, "type": "consumes" }, { - "source": 71, - "target": 72, + "source": 72, + "target": 73, "type": "consumes" }, { "source": 20, - "target": 71, + "target": 72, "type": "produces" }, { - "source": 73, - "target": 71, + "source": 74, + "target": 72, "type": "produces" }, { - "source": 74, + "source": 75, "target": 2, "type": "consumes" }, { - "source": 74, + "source": 75, "target": 20, "type": "consumes" }, { "source": 10, - "target": 74, + "target": 75, "type": "produces" }, { - "source": 75, + "source": 76, "target": 16, "type": "consumes" }, { "source": 42, - "target": 75, + "target": 76, "type": "produces" }, { "source": 20, - "target": 75, + "target": 76, "type": "produces" }, { - "source": 76, + "source": 77, "target": 7, "type": "consumes" }, { "source": 7, - "target": 76, + "target": 77, "type": "produces" }, { - "source": 77, + "source": 78, "target": 3, "type": "consumes" }, { "source": 4, - "target": 77, + "target": 78, "type": "produces" }, { - "source": 78, + "source": 79, "target": 3, "type": "consumes" }, { "source": 46, - "target": 78, + "target": 79, "type": "produces" }, { "source": 4, - "target": 78, + "target": 79, "type": "produces" }, { - "source": 79, + "source": 80, "target": 46, "type": "consumes" }, { "source": 10, - "target": 79, + "target": 80, "type": "produces" }, { - "source": 80, + "source": 81, "target": 46, "type": "consumes" }, { "source": 10, - "target": 80, + "target": 81, "type": "produces" }, { - "source": 81, + "source": 82, "target": 7, "type": "consumes" }, { "source": 46, - "target": 81, + "target": 82, "type": "produces" }, { "source": 20, - "target": 81, + "target": 82, "type": "produces" }, { - "source": 82, - "target": 66, + "source": 83, + "target": 67, "type": "consumes" }, { - "source": 82, - "target": 67, + "source": 83, + "target": 68, "type": "consumes" }, { "source": 46, - "target": 82, + "target": 83, "type": "produces" }, { - "source": 83, + "source": 84, "target": 7, "type": "consumes" }, { "source": 48, - "target": 83, + "target": 84, "type": "produces" }, { - "source": 67, - "target": 83, + "source": 68, + "target": 84, "type": "produces" }, { - "source": 84, + "source": 85, "target": 46, "type": "consumes" }, { "source": 10, - "target": 84, + "target": 85, "type": "produces" }, { - "source": 85, - "target": 67, + "source": 86, + "target": 68, "type": "consumes" }, { "source": 46, - "target": 85, + "target": 86, "type": "produces" }, { - "source": 86, + "source": 87, "target": 2, "type": "consumes" }, { - "source": 86, - "target": 67, + "source": 87, + "target": 68, "type": "consumes" }, { - "source": 86, + "source": 87, "target": 5, "type": "consumes" }, { "source": 46, - "target": 86, + "target": 87, "type": "produces" }, { "source": 4, - "target": 86, + "target": 87, "type": "produces" }, { - "source": 67, - "target": 86, + "source": 68, + "target": 87, "type": "produces" }, { "source": 5, - "target": 86, + "target": 87, "type": "produces" }, { - "source": 87, + "source": 88, "target": 46, "type": "consumes" }, { - "source": 87, - "target": 66, + "source": 88, + "target": 67, "type": "consumes" }, { "source": 9, - "target": 87, + "target": 88, "type": "produces" }, { - "source": 88, - "target": 67, + "source": 89, + "target": 68, "type": "consumes" }, { - "source": 88, + "source": 89, "target": 3, "type": "consumes" }, { "source": 5, - "target": 88, + "target": 89, "type": "produces" }, { "source": 3, - "target": 88, + "target": 89, "type": "produces" }, { "source": 20, - "target": 88, + "target": 89, "type": "produces" }, { - "source": 89, - "target": 88, + "source": 90, + "target": 89, "type": "produces" }, { - "source": 90, + "source": 91, "target": 3, "type": "consumes" }, { "source": 4, - "target": 90, + "target": 91, "type": "produces" }, { - "source": 91, + "source": 92, "target": 7, "type": "consumes" }, { "source": 7, - "target": 91, + "target": 92, "type": "produces" }, { - "source": 92, + "source": 93, "target": 2, "type": "consumes" }, { "source": 4, - "target": 92, + "target": 93, "type": "produces" }, { - "source": 93, + "source": 94, "target": 16, "type": "consumes" }, { - "source": 93, + "source": 94, "target": 3, "type": "consumes" }, { - "source": 93, + "source": 94, "target": 20, "type": "consumes" }, { "source": 2, - "target": 93, + "target": 94, "type": "produces" }, { "source": 3, - "target": 93, + "target": 94, "type": "produces" }, { - "source": 94, - "target": 73, + "source": 95, + "target": 74, "type": "consumes" }, { "source": 4, - "target": 94, + "target": 95, "type": "produces" }, { - "source": 95, + "source": 96, "target": 7, "type": "consumes" }, { "source": 7, - "target": 95, + "target": 96, "type": "produces" }, { "source": 48, - "target": 95, + "target": 96, "type": "produces" }, { "source": 20, - "target": 95, + "target": 96, "type": "produces" }, { - "source": 96, + "source": 97, "target": 3, "type": "consumes" }, { - "source": 97, - "target": 96, + "source": 98, + "target": 97, "type": "produces" }, { - "source": 98, + "source": 99, "target": 12, "type": "consumes" }, { - "source": 99, - "target": 98, + "source": 100, + "target": 99, "type": "produces" }, { - "source": 100, + "source": 101, "target": 12, "type": "consumes" }, { "source": 12, - "target": 100, + "target": 101, "type": "produces" }, { - "source": 101, + "source": 102, "target": 12, "type": "consumes" }, { - "source": 99, - "target": 101, + "source": 100, + "target": 102, "type": "produces" }, { - "source": 102, + "source": 103, "target": 10, "type": "consumes" }, { "source": 10, - "target": 102, + "target": 103, "type": "produces" }, { - "source": 103, + "source": 104, "target": 10, "type": "consumes" }, { - "source": 72, - "target": 103, + "source": 73, + "target": 104, "type": "produces" }, { - "source": 104, + "source": 105, "target": 7, "type": "consumes" }, { "source": 7, - "target": 104, + "target": 105, "type": "produces" }, { - "source": 105, + "source": 106, "target": 42, "type": "consumes" }, { "source": 4, - "target": 105, + "target": 106, "type": "produces" }, { - "source": 106, + "source": 107, "target": 3, "type": "consumes" }, { - "source": 106, - "target": 73, + "source": 107, + "target": 74, "type": "consumes" }, { "source": 4, - "target": 106, + "target": 107, "type": "produces" }, { - "source": 107, + "source": 2, + "target": 107, + "type": "produces" + }, + { + "source": 108, "target": 42, "type": "consumes" }, { "source": 4, - "target": 107, + "target": 108, "type": "produces" }, { - "source": 108, + "source": 109, "target": 7, "type": "consumes" }, { "source": 7, - "target": 108, + "target": 109, "type": "produces" }, { - "source": 109, + "source": 110, "target": 2, "type": "consumes" }, { "source": 4, - "target": 109, + "target": 110, "type": "produces" }, { - "source": 110, + "source": 111, "target": 7, "type": "consumes" }, { - "source": 110, + "source": 111, "target": 2, "type": "consumes" }, { - "source": 110, + "source": 111, "target": 12, "type": "consumes" }, { - "source": 110, + "source": 111, "target": 16, "type": "consumes" }, { - "source": 110, + "source": 111, "target": 42, "type": "consumes" }, { - "source": 111, + "source": 112, "target": 2, "type": "consumes" }, { - "source": 111, + "source": 112, "target": 3, "type": "consumes" }, { "source": 7, - "target": 111, + "target": 112, "type": "produces" }, { "source": 4, - "target": 111, + "target": 112, "type": "produces" }, { - "source": 112, + "source": 113, "target": 3, "type": "consumes" }, { "source": 4, - "target": 112, + "target": 113, "type": "produces" }, { "source": 5, - "target": 112, + "target": 113, "type": "produces" }, { - "source": 113, + "source": 114, "target": 7, "type": "consumes" }, { - "source": 113, + "source": 114, "target": 20, "type": "consumes" }, { "source": 7, - "target": 113, + "target": 114, "type": "produces" }, { - "source": 114, + "source": 115, "target": 7, "type": "consumes" }, { "source": 7, - "target": 114, + "target": 115, "type": "produces" }, { - "source": 115, + "source": 116, "target": 2, "type": "consumes" }, { - "source": 115, - "target": 73, + "source": 116, + "target": 74, "type": "consumes" }, { - "source": 73, - "target": 115, + "source": 74, + "target": 116, "type": "produces" }, { - "source": 116, + "source": 117, "target": 2, "type": "consumes" }, { - "source": 116, - "target": 73, + "source": 117, + "target": 74, "type": "consumes" }, { - "source": 73, - "target": 116, + "source": 74, + "target": 117, "type": "produces" }, { - "source": 117, + "source": 118, "target": 2, "type": "consumes" }, { - "source": 117, - "target": 73, + "source": 118, + "target": 74, "type": "consumes" }, { - "source": 73, - "target": 117, + "source": 74, + "target": 118, "type": "produces" }, { - "source": 118, + "source": 119, "target": 7, "type": "consumes" }, { "source": 48, - "target": 118, + "target": 119, "type": "produces" }, { - "source": 119, + "source": 120, "target": 16, "type": "consumes" }, { - "source": 119, + "source": 120, "target": 3, "type": "consumes" }, { - "source": 119, + "source": 120, "target": 20, "type": "consumes" }, { - "source": 120, + "source": 121, "target": 7, "type": "consumes" }, { - "source": 120, + "source": 121, "target": 12, "type": "consumes" }, { - "source": 120, - "target": 121, + "source": 121, + "target": 63, "type": "consumes" }, { "source": 16, - "target": 120, + "target": 121, "type": "produces" }, { "source": 122, - "target": 66, + "target": 67, "type": "consumes" }, { "source": 122, - "target": 67, + "target": 68, "type": "consumes" }, { @@ -1371,7 +1381,7 @@ }, { "source": 125, - "target": 73, + "target": 74, "type": "consumes" }, { @@ -1505,7 +1515,7 @@ "type": "consumes" }, { - "source": 67, + "source": 68, "target": 134, "type": "produces" }, @@ -1536,12 +1546,12 @@ }, { "source": 135, - "target": 121, + "target": 63, "type": "consumes" }, { "source": 135, - "target": 67, + "target": 68, "type": "consumes" }, { @@ -1585,7 +1595,7 @@ "type": "produces" }, { - "source": 66, + "source": 67, "target": 135, "type": "produces" }, @@ -1696,7 +1706,7 @@ }, { "source": 143, - "target": 72, + "target": 73, "type": "consumes" }, { @@ -1751,102 +1761,127 @@ }, { "source": 148, - "target": 7, + "target": 3, "type": "consumes" }, { - "source": 7, + "source": 149, "target": 148, "type": "produces" }, { - "source": 149, - "target": 3, - "type": "consumes" + "source": 2, + "target": 148, + "type": "produces" }, { - "source": 17, - "target": 149, + "source": 150, + "target": 148, "type": "produces" }, { - "source": 150, + "source": 151, "target": 7, "type": "consumes" }, { "source": 7, - "target": 150, + "target": 151, "type": "produces" }, { - "source": 20, - "target": 150, + "source": 152, + "target": 3, + "type": "consumes" + }, + { + "source": 4, + "target": 152, "type": "produces" }, { - "source": 151, - "target": 73, + "source": 153, + "target": 3, "type": "consumes" }, { - "source": 152, - "target": 4, - "type": "consumes" + "source": 17, + "target": 153, + "type": "produces" }, { - "source": 152, - "target": 5, + "source": 154, + "target": 7, "type": "consumes" }, { - "source": 152, + "source": 154, "target": 3, "type": "consumes" }, { - "source": 153, - "target": 3, - "type": "consumes" + "source": 7, + "target": 154, + "type": "produces" }, { - "source": 20, - "target": 153, + "source": 4, + "target": 154, "type": "produces" }, { - "source": 154, - "target": 97, - "type": "consumes" + "source": 2, + "target": 154, + "type": "produces" }, { "source": 20, "target": 154, "type": "produces" }, + { + "source": 74, + "target": 154, + "type": "produces" + }, { "source": 155, - "target": 2, + "target": 74, "type": "consumes" }, { - "source": 155, + "source": 156, + "target": 4, + "type": "consumes" + }, + { + "source": 156, "target": 5, "type": "consumes" }, { - "source": 4, - "target": 155, - "type": "produces" + "source": 156, + "target": 3, + "type": "consumes" }, { - "source": 5, - "target": 155, + "source": 157, + "target": 3, + "type": "consumes" + }, + { + "source": 20, + "target": 157, "type": "produces" }, + { + "source": 158, + "target": 98, + "type": "consumes" + }, { "source": 20, - "target": 155, + "target": 158, "type": "produces" } ] \ No newline at end of file diff --git a/docs/modules/list_of_modules.md b/docs/modules/list_of_modules.md index 520856ac03..7c71adaf4b 100644 --- a/docs/modules/list_of_modules.md +++ b/docs/modules/list_of_modules.md @@ -50,10 +50,11 @@ | sslcert | scan | No | Extract hostnames and emails from TLS certificates in HTTP responses | active, affiliates, email-enum, safe, subdomain-enum, web | HTTP_RESPONSE | DNS_NAME, EMAIL_ADDRESS | @TheTechromancer | 2022-03-30 | | telerik | scan | No | Scan for critical Telerik vulnerabilities | active, invasive, loud, web-heavy | HTTP_RESPONSE, URL | FINDING | @liquidsec | 2022-04-10 | | url_manipulation | scan | No | Attempt to identify URL parsing/routing based vulnerabilities | active, loud, web-heavy | URL | FINDING | @liquidsec | 2022-09-27 | +| virtualhost | scan | No | Fuzz for virtual hosts | active, loud, slow | URL | DNS_NAME_UNVERIFIED, HTTP_RESPONSE, VIRTUAL_HOST | @liquidsec | 2022-05-02 | +| waf_bypass | scan | No | Detects potential WAF bypasses | active, safe, web-heavy | URL | FINDING | @liquidsec | 2025-09-26 | | wafw00f | scan | No | Web Application Firewall Fingerprinting Tool | active, loud | URL | WAF | @liquidsec | 2023-02-15 | | webbrute | scan | No | A fast web fuzzer powered by blasthttp | active, loud | URL | URL_UNVERIFIED | @liquidsec | 2022-04-10 | | webbrute_shortnames | scan | No | Brute-force IIS shortnames using ML-predicted wordlists | active, iis-shortnames, loud, web-heavy | URL_HINT | URL_UNVERIFIED | @liquidsec | 2022-07-05 | -| wpscan | scan | No | Wordpress security scanner. Highly recommended to use an API key for better results. | active, loud | HTTP_RESPONSE, TECHNOLOGY | FINDING, TECHNOLOGY, URL_UNVERIFIED | @domwhewell-sage | 2024-05-29 | | affiliates | scan | No | Summarize affiliate domains at the end of a scan | affiliates, passive, safe | * | | @TheTechromancer | 2022-07-25 | | anubisdb | scan | No | Query anubisdb.com for subdomains | passive, safe, subdomain-enum | DNS_NAME | DNS_NAME | @TheTechromancer | 2022-10-04 | | apkpure | scan | No | Download android applications from apkpure.com | code-enum, download, passive, safe | MOBILE_APP | FILESYSTEM | @domwhewell-sage | 2024-10-11 | @@ -117,7 +118,7 @@ | urlscan | scan | No | Query urlscan.io for subdomains | passive, safe, subdomain-enum | DNS_NAME | DNS_NAME, URL_UNVERIFIED | @TheTechromancer | 2022-06-09 | | viewdns | scan | No | Query viewdns.info's reverse whois for related domains | affiliates, passive, safe | DNS_NAME | DNS_NAME | @TheTechromancer | 2022-07-04 | | virustotal | scan | Yes | Query VirusTotal's API for subdomains | passive, safe, subdomain-enum | DNS_NAME | DNS_NAME | @TheTechromancer | 2022-08-25 | -| wayback | scan | No | Query archive.org's API for subdomains | passive, safe, subdomain-enum | DNS_NAME | DNS_NAME, URL_UNVERIFIED | @liquidsec | 2022-04-01 | +| wayback | scan | No | Query archive.org's Wayback Machine for subdomains, URLs, parameters, and archived content | passive, safe, subdomain-enum | DNS_NAME, URL | DNS_NAME, FINDING, HTTP_RESPONSE, URL_UNVERIFIED, WEB_PARAMETER | @liquidsec | 2022-04-01 | | asset_inventory | output | No | Merge hosts, open ports, technologies, findings, etc. into a single asset inventory CSV | | DNS_NAME, FINDING, HTTP_RESPONSE, IP_ADDRESS, OPEN_TCP_PORT, TECHNOLOGY, URL, WAF | IP_ADDRESS, OPEN_TCP_PORT | @liquidsec | 2022-09-30 | | csv | output | No | Output to CSV | | * | | @TheTechromancer | 2022-04-07 | | discord | output | No | Message a Discord channel when certain events are encountered | | * | | @TheTechromancer | 2023-08-14 | @@ -131,7 +132,6 @@ | neo4j | output | No | Output to Neo4j | | * | | @TheTechromancer | 2022-04-07 | | nmap_xml | output | No | Output to Nmap XML | | DNS_NAME, HTTP_RESPONSE, IP_ADDRESS, OPEN_TCP_PORT, PROTOCOL | | @TheTechromancer | 2024-11-16 | | postgres | output | No | Output scan data to a SQLite database | | * | | @TheTechromancer | 2024-11-08 | -| python | output | No | Output via Python API | | * | | @TheTechromancer | 2022-09-13 | | rabbitmq | output | No | Output scan data to a RabbitMQ queue | | * | | @TheTechromancer | 2024-11-22 | | slack | output | No | Message a Slack channel when certain events are encountered | | * | | @TheTechromancer | 2023-08-14 | | splunk | output | No | Send every event to a splunk instance through HTTP Event Collector | | * | | @w0Tx | 2024-02-17 | @@ -146,7 +146,8 @@ | websocket | output | No | Output to websockets | | * | | @TheTechromancer | 2022-04-15 | | zeromq | output | No | Output scan data to a ZeroMQ socket (PUB) | | * | | @TheTechromancer | 2024-11-22 | | cloudcheck | internal | No | Tag events by cloud provider, identify cloud resources like storage buckets | | * | | @TheTechromancer | 2024-07-07 | -| dnsresolve | internal | No | Perform DNS resolution | | * | DNS_NAME, IP_ADDRESS, RAW_DNS_RECORD | @TheTechromancer | 2022-04-08 | +| dnsresolve | internal | No | Perform DNS resolution | | * | DNS_NAME, IP_ADDRESS, IP_RANGE, RAW_DNS_RECORD | @TheTechromancer | 2022-04-08 | +| python | internal | No | Output via Python API | | * | | @TheTechromancer | 2022-09-13 | | aggregate | internal | No | Summarize statistics at the end of a scan | passive, safe | | | @TheTechromancer | 2022-07-25 | | excavate | internal | No | Passively extract juicy tidbits from scan data | passive, safe | HTTP_RESPONSE, RAW_TEXT | URL_UNVERIFIED, WEB_PARAMETER | @liquidsec | 2022-06-27 | | speculate | internal | No | Derive certain event types from others by common sense | passive, safe | AZURE_TENANT, DNS_NAME, DNS_NAME_UNRESOLVED, HTTP_RESPONSE, IP_ADDRESS, IP_RANGE, SOCIAL, STORAGE_BUCKET, URL, URL_UNVERIFIED, USERNAME | DNS_NAME, FINDING, IP_ADDRESS, OPEN_TCP_PORT, ORG_STUB | @liquidsec | 2022-05-03 | diff --git a/docs/modules/nuclei.md b/docs/modules/nuclei.md index aefc4e1d4f..c51e041067 100644 --- a/docs/modules/nuclei.md +++ b/docs/modules/nuclei.md @@ -36,22 +36,22 @@ modules: The Nuclei module has many configuration options: <!-- BBOT MODULE OPTIONS NUCLEI --> -| Config Option | Type | Description | Default | -|-------------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| -| modules.nuclei.batch_size | int | Number of targets to send to Nuclei per batch (default 200) | 200 | -| modules.nuclei.budget | int | Used in budget mode to set the number of allowed requests per host | 1 | -| modules.nuclei.concurrency | int | maximum number of templates to be executed in parallel (default 25) | 25 | -| modules.nuclei.directory_only | bool | Filter out 'file' URL event (default True) | True | -| modules.nuclei.etags | str | tags to exclude from the scan | | -| modules.nuclei.mode | str | manual | technology | severe | budget. Technology: Only activate based on technology events that match nuclei tags (nuclei -as mode). Manual (DEFAULT): Fully manual settings. Severe: Only critical and high severity templates without intrusive. Budget: Limit Nuclei to a specified number of HTTP requests | manual | -| modules.nuclei.module_timeout | int | Max time in seconds to spend handling each batch of events | 21600 | -| modules.nuclei.ratelimit | int | maximum number of requests to send per second (default 150) | 150 | -| modules.nuclei.retries | int | number of times to retry a failed request (default 0) | 0 | -| modules.nuclei.severity | str | Filter based on severity field available in the template. | | -| modules.nuclei.silent | bool | Don't display nuclei's banner or status messages | False | -| modules.nuclei.tags | str | execute a subset of templates that contain the provided tags | | -| modules.nuclei.templates | str | template or template directory paths to include in the scan | | -| modules.nuclei.version | str | nuclei version | 3.8.0 | +| Config Option | Type | Description | Default | +|-------------------------------|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| +| modules.nuclei.batch_size | int | Number of targets to send to Nuclei per batch (default 200) | 200 | +| modules.nuclei.budget | int | Used in budget mode to set the number of allowed requests per host | 1 | +| modules.nuclei.concurrency | int | maximum number of templates to be executed in parallel (default 25) | 25 | +| modules.nuclei.directory_only | bool | Filter out 'file' URL event (default True) | True | +| modules.nuclei.etags | str | tags to exclude from the scan | | +| modules.nuclei.mode | Literal['manual', 'technology', 'severe', 'budget'] | manual | technology | severe | budget. Technology: Only activate based on technology events that match nuclei tags (nuclei -as mode). Manual (DEFAULT): Fully manual settings. Severe: Only critical and high severity templates without intrusive. Budget: Limit Nuclei to a specified number of HTTP requests | manual | +| modules.nuclei.module_timeout | int | Max time in seconds to spend handling each batch of events | 21600 | +| modules.nuclei.ratelimit | int | maximum number of requests to send per second (default 150) | 150 | +| modules.nuclei.retries | int | number of times to retry a failed request (default 0) | 0 | +| modules.nuclei.severity | str | Filter based on severity field available in the template. | | +| modules.nuclei.silent | bool | Don't display nuclei's banner or status messages | False | +| modules.nuclei.tags | str | execute a subset of templates that contain the provided tags | | +| modules.nuclei.templates | str | template or template directory paths to include in the scan | | +| modules.nuclei.version | str | nuclei version | 3.9.0 | <!-- END BBOT MODULE OPTIONS NUCLEI --> Most of these you probably will **NOT** want to change. In particular, we advise against changing the version of Nuclei, as it's possible the latest version won't work right with BBOT. diff --git a/docs/scanning/advanced.md b/docs/scanning/advanced.md index 3aa08c9ad2..7f8dd9c089 100644 --- a/docs/scanning/advanced.md +++ b/docs/scanning/advanced.md @@ -40,10 +40,12 @@ usage: bbot [-h] [-t TARGET [TARGET ...]] [-s SEEDS [SEEDS ...]] [-ef FLAG [FLAG ...]] [-n SCAN_NAME] [-v] [-d] [-S] [--force] [-y] [--fast-mode] [--dry-run] [--current-preset] [--current-preset-full] [-mh MODULE] [-o DIR] - [-om MODULE [MODULE ...]] [-lo] [--json] [--brief] [--no-color] + [-om MODULE [MODULE ...]] [-eom MODULE [MODULE ...]] [-lo] + [--json] [--brief] [--no-color] [--event-types EVENT_TYPES [EVENT_TYPES ...]] [--exclude-cdn] - [--no-deps | --force-deps | --retry-deps | --ignore-failed-deps] - [--install-all-deps] [--version] [--proxy HTTP_PROXY] + [--no-deps | --force-deps | --retry-deps | + --ignore-failed-deps] [--install-all-deps] [--version] + [--reset-config] [--reset-secrets] [--proxy HTTP_PROXY] [--no-proxy HOST [HOST ...]] [-H CUSTOM_HEADERS [CUSTOM_HEADERS ...]] [-C CUSTOM_COOKIES [CUSTOM_COOKIES ...]] @@ -56,40 +58,39 @@ options: -h, --help show this help message and exit Target: - -t TARGET [TARGET ...], --targets TARGET [TARGET ...] + -t, --targets TARGET [TARGET ...] Target scope - -s SEEDS [SEEDS ...], --seeds SEEDS [SEEDS ...] + -s, --seeds SEEDS [SEEDS ...] Define seeds to drive passive modules without being in scope (if not specified, defaults to same as targets) - -b BLACKLIST [BLACKLIST ...], --blacklist BLACKLIST [BLACKLIST ...] + -b, --blacklist BLACKLIST [BLACKLIST ...] Don't touch these things --strict-scope Don't consider subdomains of target to be in-scope - exact matches only Presets: - -p [PRESET ...], --preset [PRESET ...] + -p, --preset [PRESET ...] Enable BBOT preset(s) - -c [CONFIG ...], --config [CONFIG ...] + -c, --config [CONFIG ...] Custom config options in key=value format: e.g. 'modules.shodan.api_key=1234' -lp, --list-presets List available presets. Modules: - -m MODULE [MODULE ...], --modules MODULE [MODULE ...] - Modules to enable. Choices: affiliates,ajaxpro,anubisdb,apkpure,asn,aspnet_bin_exposure,azure_tenant,baddns,baddns_direct,baddns_zone,badsecrets,bevigil,bucket_amazon,bucket_digitalocean,bucket_file_enum,bucket_firebase,bucket_google,bucket_hetzner,bucket_microsoft,bufferoverrun,builtwith,bypass403,c99,censys_dns,censys_ip,certspotter,chaos,code_repository,credshed,crt,crt_db,dehashed,dnsbimi,dnsbrute,dnsbrute_mutations,dnscaa,dnscommonsrv,dnsdumpster,dnstlsrpt,docker_pull,dockerhub,dotnetnuke,emailformat,filedownload,fingerprintx,fullhunt,generic_ssrf,git,git_clone,gitdumper,github_codesearch,github_org,github_usersearch,github_workflows,gitlab_com,gitlab_onprem,google_playstore,gowitness,graphql_introspection,hackertarget,host_header,http,hunt,hunterio,iis_shortnames,ip2location,ipneighbor,ipstack,jadx,kreuzberg,leakix,legba,lightfuzz,medusa,myssl,newsletters,ntlm,nuclei,oauth,otx,paramminer_cookies,paramminer_getparams,paramminer_headers,pgp,portfilter,portscan,postman,postman_download,rapiddns,reflected_parameters,retirejs,robots,securitytrails,securitytxt,shodan_dns,shodan_enterprise,shodan_idb,skymem,social,sslcert,subdomaincenter,subdomainradar,telerik,trajan,trickest,trufflehog,url_manipulation,urlscan,viewdns,virustotal,wafw00f,wayback,webbrute,webbrute_shortnames,wpscan + -m, --modules MODULE [MODULE ...] + Modules to enable. Choices: affiliates,ajaxpro,anubisdb,apkpure,asn,aspnet_bin_exposure,azure_tenant,baddns,baddns_direct,baddns_zone,badsecrets,bevigil,bucket_amazon,bucket_digitalocean,bucket_file_enum,bucket_firebase,bucket_google,bucket_hetzner,bucket_microsoft,bufferoverrun,builtwith,bypass403,c99,censys_dns,censys_ip,certspotter,chaos,code_repository,credshed,crt,crt_db,dehashed,dnsbimi,dnsbrute,dnsbrute_mutations,dnscaa,dnscommonsrv,dnsdumpster,dnstlsrpt,docker_pull,dockerhub,dotnetnuke,emailformat,filedownload,fingerprintx,fullhunt,generic_ssrf,git,git_clone,gitdumper,github_codesearch,github_org,github_usersearch,github_workflows,gitlab_com,gitlab_onprem,google_playstore,gowitness,graphql_introspection,hackertarget,host_header,http,hunt,hunterio,iis_shortnames,ip2location,ipneighbor,ipstack,jadx,kreuzberg,leakix,legba,lightfuzz,medusa,myssl,newsletters,ntlm,nuclei,oauth,otx,paramminer_cookies,paramminer_getparams,paramminer_headers,pgp,portfilter,portscan,postman,postman_download,rapiddns,reflected_parameters,retirejs,robots,securitytrails,securitytxt,shodan_dns,shodan_enterprise,shodan_idb,skymem,social,sslcert,subdomaincenter,subdomainradar,telerik,trajan,trickest,trufflehog,url_manipulation,urlscan,viewdns,virtualhost,virustotal,waf_bypass,wafw00f,wayback,webbrute,webbrute_shortnames -l, --list-modules List available modules. -lmo, --list-module-options Show all module config options - -em MODULE [MODULE ...], --exclude-modules MODULE [MODULE ...] + -em, --exclude-modules MODULE [MODULE ...] Exclude these modules. - -f FLAG [FLAG ...], --flags FLAG [FLAG ...] + -f, --flags FLAG [FLAG ...] Enable modules by flag. Choices: active,affiliates,baddns,cloud-enum,code-enum,download,email-enum,iis-shortnames,invasive,loud,passive,portscan,safe,service-enum,slow,social-enum,subdomain-enum,subdomain-hijack,web,web-heavy,web-paramminer,web-screenshots -lf, --list-flags List available flags. - -rf FLAG [FLAG ...], --require-flags FLAG [FLAG ...] + -rf, --require-flags FLAG [FLAG ...] Only enable modules with these flags (e.g. -rf passive) - -ef FLAG [FLAG ...], --exclude-flags FLAG [FLAG ...] + -ef, --exclude-flags FLAG [FLAG ...] Disable modules with these flags. (e.g. -ef loud) Scan: - -n SCAN_NAME, --name SCAN_NAME - Name of scan (default: random) + -n, --name SCAN_NAME Name of scan (default: random) -v, --verbose Be more verbose -d, --debug Enable debugging -S, --silent Be quiet @@ -100,14 +101,15 @@ Scan: --current-preset Show the current preset in YAML format --current-preset-full Show the current preset in its full form, including defaults - -mh MODULE, --module-help MODULE + -mh, --module-help MODULE Show help for a specific module Output: - -o DIR, --output-dir DIR - Directory to output scan results - -om MODULE [MODULE ...], --output-modules MODULE [MODULE ...] - Output module(s). Choices: asset_inventory,csv,discord,elastic,emails,json,kafka,mongo,mysql,nats,neo4j,nmap_xml,postgres,python,rabbitmq,slack,splunk,sqlite,stdout,subdomains,teams,txt,web_parameters,web_report,webhook,websocket,zeromq + -o, --output-dir DIR Directory to output scan results + -om, --output-modules MODULE [MODULE ...] + Add output module(s). Choices: asset_inventory,csv,discord,elastic,emails,json,kafka,mongo,mysql,nats,neo4j,nmap_xml,postgres,rabbitmq,slack,splunk,sqlite,stdout,subdomains,teams,txt,web_parameters,web_report,webhook,websocket,zeromq + -eom, --exclude-output-modules MODULE [MODULE ...] + Exclude output module(s) -lo, --list-output-modules List available output modules --json, -j Output scan data in JSON format @@ -128,18 +130,20 @@ Module dependencies: Misc: --version show BBOT version and exit + --reset-config Regenerate bbot.yml from current defaults (overwrites; backs up to .bak) + --reset-secrets Regenerate secrets.yml from current defaults (overwrites; backs up to .bak) --proxy HTTP_PROXY Use this proxy for all HTTP requests --no-proxy HOST [HOST ...] Exclude these hosts from proxy (e.g. localhost *.internal.corp 10.0.0.0/8) - -H CUSTOM_HEADERS [CUSTOM_HEADERS ...], --custom-headers CUSTOM_HEADERS [CUSTOM_HEADERS ...] + -H, --custom-headers CUSTOM_HEADERS [CUSTOM_HEADERS ...] List of custom headers as key value pairs (header=value). - -C CUSTOM_COOKIES [CUSTOM_COOKIES ...], --custom-cookies CUSTOM_COOKIES [CUSTOM_COOKIES ...] + -C, --custom-cookies CUSTOM_COOKIES [CUSTOM_COOKIES ...] List of custom cookies as key value pairs (cookie=value). - --custom-yara-rules CUSTOM_YARA_RULES, -cy CUSTOM_YARA_RULES + --custom-yara-rules, -cy CUSTOM_YARA_RULES Add custom yara rules to excavate - --user-agent USER_AGENT, -ua USER_AGENT + --user-agent, -ua USER_AGENT Set the user-agent for all HTTP requests - --user-agent-suffix SUFFIX, -uas SUFFIX + --user-agent-suffix, -uas SUFFIX Suffix to append to the user-agent EXAMPLES diff --git a/docs/scanning/configuration.md b/docs/scanning/configuration.md index 6bdacb6db3..6c42a0ab39 100644 --- a/docs/scanning/configuration.md +++ b/docs/scanning/configuration.md @@ -78,6 +78,8 @@ status_frequency: 15 # that scales linearly from 0s at the threshold up to 5s at threshold+5 (capped at 95%). # Last-ditch effort to give the pipeline a chance to drain before OOM. max_mem_percent: 90 +# Redact secrets (API keys, tokens, etc.) in the saved preset.yml +redact_secrets: true # Include the raw data of files (i.e. PDFs, web screenshots) as base64 in the event file_blobs: false # Include the raw data of directories (i.e. git repos) as tar.gz base64 in the event @@ -241,6 +243,8 @@ aggregate: True dnsresolve: True # Cloud provider tagging cloudcheck: True +# Python API event bridge +python: True # Strip querystring from URLs by default url_querystring_remove: True @@ -342,12 +346,29 @@ parameter_blacklist: - PHPSESSID - sessionid - csrftoken + - XSRF-TOKEN - __cf_bm + - _cfuvid - cf_clearance - _abck - bm_sz + - bm_sv - ak_bmsc - f5_cspm + # CSRF / Anti-Forgery tokens + - authenticity_token + - csrfmiddlewaretoken + - __RequestVerificationToken + - antiforgerytoken + - __csrf_magic + - _wpnonce + # ASP.NET session/identity cookies + - .ASPXANONYMOUS + - .ASPXAUTH + # PKCE (Proof Key for Code Exchange) + - code_verifier + - code_challenge + # Analytics - _ga - _gid - _gat @@ -366,6 +387,7 @@ parameter_blacklist_prefixes: - f5avr - incap_ - visid_incap_ + - nlbi_ - AWSALB - utm_ - ApplicationGatewayAffinity @@ -374,6 +396,7 @@ parameter_blacklist_prefixes: - _hjSession - _gat_ - intercom- + - OAMRequestContext_ # Don't output these types of events (they are still distributed to modules) omit_event_types: @@ -416,324 +439,337 @@ In addition to the stated options for each module, the following universal optio ### Module Options <!-- BBOT MODULE OPTIONS --> -| Config Option | Type | Description | Default | -|-----------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| modules.baddns.custom_nameservers | list | Force BadDNS to use a list of custom nameservers | [] | -| modules.baddns.enabled_submodules | list | A list of submodules to enable. Empty list (default) enables CNAME, TXT and MX Only | [] | -| modules.baddns.min_confidence | str | Minimum confidence to emit | MEDIUM | -| modules.baddns.min_severity | str | Minimum severity to emit | LOW | -| modules.baddns_direct.custom_nameservers | list | Force BadDNS to use a list of custom nameservers | [] | -| modules.baddns_direct.min_confidence | str | Minimum confidence to emit (UNKNOWN, LOW, MEDIUM, HIGH, CONFIRMED) | MEDIUM | -| modules.baddns_direct.min_severity | str | Minimum severity to emit (INFO, LOW, MEDIUM, HIGH, CRITICAL) | LOW | -| modules.baddns_zone.custom_nameservers | list | Force BadDNS to use a list of custom nameservers | [] | -| modules.baddns_zone.min_confidence | str | Minimum confidence to emit (UNKNOWN, LOW, MEDIUM, HIGH, CONFIRMED) | MEDIUM | -| modules.baddns_zone.min_severity | str | Minimum severity to emit (INFO, LOW, MEDIUM, HIGH, CRITICAL) | INFO | -| modules.badsecrets.custom_secrets | NoneType | Include custom secrets loaded from a local file | None | -| modules.bucket_amazon.permutations | bool | Whether to try permutations | False | -| modules.bucket_digitalocean.permutations | bool | Whether to try permutations | False | -| modules.bucket_firebase.permutations | bool | Whether to try permutations | False | -| modules.bucket_google.permutations | bool | Whether to try permutations | False | -| modules.bucket_hetzner.permutations | bool | Whether to try permutations | False | -| modules.bucket_microsoft.permutations | bool | Whether to try permutations | False | -| modules.dnsbrute.max_depth | int | How many subdomains deep to brute force, i.e. 5.4.3.2.1.evilcorp.com | 5 | -| modules.dnsbrute.recursive_mutations | bool | If True, brute-force hosts discovered by dnsbrute_mutations. The default (False) skips them because the static wordlist heavily overlaps with the mutation algorithm's own output. | False | -| modules.dnsbrute.wordlist | str | Subdomain wordlist URL or file path. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/subdomains-top1million-5000.txt | -| modules.dnsbrute_mutations.max_mutations | int | Maximum number of target-specific mutations to try per subdomain | 100 | -| modules.dnscommonsrv.max_depth | int | The maximum subdomain depth to brute-force SRV records | 2 | -| modules.dnscommonsrv.recursive_mutations | bool | If True, brute-force SRV records on hosts discovered by dnsbrute_mutations. Default False skips them. | False | -| modules.filedownload.extensions | list | File extensions to download | ['bak', 'bash', 'bashrc', 'cfg', 'conf', 'crt', 'csv', 'db', 'dll', 'doc', 'docx', 'exe', 'ica', 'indd', 'ini', 'jar', 'json', 'key', 'log', 'markdown', 'md', 'msi', 'odg', 'odp', 'ods', 'odt', 'pdf', 'pem', 'pps', 'ppsx', 'ppt', 'pptx', 'ps1', 'pub', 'raw', 'rdp', 'rsa', 'sh', 'sql', 'sqlite', 'swp', 'sxw', 'tar.gz', 'tgz', 'tar', 'txt', 'vbs', 'war', 'wpd', 'xls', 'xlsx', 'xml', 'yaml', 'yml', 'zip', 'lzma', 'rar', '7z', 'xz', 'bz2'] | -| modules.filedownload.max_filesize | str | Cancel download if filesize is greater than this size | 10MB | -| modules.filedownload.output_folder | str | Folder to download files to. If not specified, downloaded files will be deleted when the scan completes, to minimize disk usage. | | -| modules.fingerprintx.skip_common_web | bool | Skip common web ports such as 80, 443, 8080, 8443, etc. | True | -| modules.fingerprintx.version | str | fingerprintx version | 1.1.4 | -| modules.generic_ssrf.skip_dns_interaction | bool | Do not report DNS interactions (only HTTP interaction) | False | -| modules.gitlab_com.api_key | str | GitLab access token (for gitlab.com/org only) | | -| modules.gitlab_onprem.api_key | str | GitLab access token (for self-hosted instances only) | | -| modules.gowitness.chrome_path | str | Path to chrome executable | | -| modules.gowitness.idle_timeout | int | Skip the current gowitness batch if it stalls for longer than this many seconds | 1800 | -| modules.gowitness.output_path | str | Where to save screenshots | | -| modules.gowitness.resolution_x | int | Screenshot resolution x | 1440 | -| modules.gowitness.resolution_y | int | Screenshot resolution y | 900 | -| modules.gowitness.social | bool | Whether to screenshot social media webpages | False | -| modules.gowitness.threads | int | How many gowitness threads to spawn (default is number of CPUs x 2) | 0 | -| modules.gowitness.timeout | int | Preflight check timeout | 10 | -| modules.gowitness.version | str | Gowitness version | 3.1.1 | -| modules.graphql_introspection.graphql_endpoint_urls | list | List of GraphQL endpoint to suffix to the target URL | ['/', '/graphql', '/v1/graphql'] | -| modules.graphql_introspection.output_folder | str | Folder to save the GraphQL schemas to | | -| modules.http.in_scope_only | bool | Only visit web resources that are in scope. | True | -| modules.http.max_response_size | int | Max response size in bytes | 5242880 | -| modules.http.store_responses | bool | Save raw HTTP responses to scan folder | False | -| modules.http.threads | int | Number of concurrent requests | 50 | -| modules.iis_shortnames.detect_only | bool | Only detect the vulnerability and do not run the shortname scanner | True | -| modules.iis_shortnames.max_node_count | int | Limit how many nodes to attempt to resolve on any given recursion branch | 50 | -| modules.iis_shortnames.speculate_magic_urls | bool | Attempt to discover iis 'magic' special folders | True | -| modules.legba.concurrency | int | Number of concurrent workers, gets overridden for SSH | 3 | -| modules.legba.ftp_wordlist | str | Wordlist for FTP combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/ftp-betterdefaultpasslist.txt | -| modules.legba.mssql_wordlist | str | Wordlist for MSSQL combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/mssql-betterdefaultpasslist.txt | -| modules.legba.mysql_wordlist | str | Wordlist for MySQL combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/mysql-betterdefaultpasslist.txt | -| modules.legba.postgresql_wordlist | str | Wordlist for PostgreSQL combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/postgres-betterdefaultpasslist.txt | -| modules.legba.rate_limit | int | Limit the number of requests per second, gets overridden for SSH | 3 | -| modules.legba.ssh_wordlist | str | Wordlist for SSH combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/ssh-betterdefaultpasslist.txt | -| modules.legba.telnet_wordlist | str | Wordlist for TELNET combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/telnet-betterdefaultpasslist.txt | -| modules.legba.version | str | legba version | 1.1.1 | -| modules.legba.vnc_wordlist | str | Wordlist for VNC passwords, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/vnc-betterdefaultpasslist.txt | -| modules.lightfuzz.avoid_wafs | bool | Avoid running against confirmed WAFs, which are likely to block lightfuzz requests | True | -| modules.lightfuzz.disable_post | bool | Disable processing of POST parameters, avoiding form submissions. | False | -| modules.lightfuzz.enabled_submodules | list | A list of submodules to enable. Empty list enabled all modules. | ['sqli', 'cmdi', 'xss', 'path', 'ssti', 'crypto', 'serial', 'esi', 'ssrf'] | -| modules.lightfuzz.force_common_headers | bool | Force emit commonly exploitable parameters that may be difficult to detect | False | -| modules.lightfuzz.try_get_as_post | bool | For each GETPARAM, also fuzz it as a POSTPARAM (in addition to normal GET fuzzing). | False | -| modules.lightfuzz.try_post_as_get | bool | For each POSTPARAM, also fuzz it as a GETPARAM (in addition to normal POST fuzzing). | False | -| modules.medusa.snmp_versions | list | List of SNMP versions to attempt against the SNMP server (default ['1', '2C']) | ['1', '2C'] | -| modules.medusa.snmp_wordlist | str | Wordlist url for SNMP community strings, newline separated (default https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Discovery/SNMP/snmp.txt). Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Discovery/SNMP/common-snmp-community-strings.txt | -| modules.medusa.threads | int | Number of communities to be tested concurrently (default 5) | 5 | -| modules.medusa.timeout_s | int | Wait time for the SNMP response(s) once at the end of all attempts (default 5) | 5 | -| modules.medusa.wait_microseconds | int | Wait time after every SNMP request in microseconds (default 200) | 200 | -| modules.ntlm.try_all | bool | Try every NTLM endpoint | False | -| modules.nuclei.batch_size | int | Number of targets to send to Nuclei per batch (default 200) | 200 | -| modules.nuclei.budget | int | Used in budget mode to set the number of allowed requests per host | 1 | -| modules.nuclei.concurrency | int | maximum number of templates to be executed in parallel (default 25) | 25 | -| modules.nuclei.directory_only | bool | Filter out 'file' URL event (default True) | True | -| modules.nuclei.etags | str | tags to exclude from the scan | | -| modules.nuclei.mode | str | manual | technology | severe | budget. Technology: Only activate based on technology events that match nuclei tags (nuclei -as mode). Manual (DEFAULT): Fully manual settings. Severe: Only critical and high severity templates without intrusive. Budget: Limit Nuclei to a specified number of HTTP requests | manual | -| modules.nuclei.module_timeout | int | Max time in seconds to spend handling each batch of events | 21600 | -| modules.nuclei.ratelimit | int | maximum number of requests to send per second (default 150) | 150 | -| modules.nuclei.retries | int | number of times to retry a failed request (default 0) | 0 | -| modules.nuclei.severity | str | Filter based on severity field available in the template. | | -| modules.nuclei.silent | bool | Don't display nuclei's banner or status messages | False | -| modules.nuclei.tags | str | execute a subset of templates that contain the provided tags | | -| modules.nuclei.templates | str | template or template directory paths to include in the scan | | -| modules.nuclei.version | str | nuclei version | 3.8.0 | -| modules.oauth.try_all | bool | Check for OAUTH/IODC on every subdomain and URL. | False | -| modules.paramminer_cookies.recycle_words | bool | Attempt to use words found during the scan on all other endpoints | False | -| modules.paramminer_cookies.skip_boring_words | bool | Remove commonly uninteresting words from the wordlist | True | -| modules.paramminer_cookies.wordlist | str | Define the wordlist to be used to derive cookies. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | -| modules.paramminer_getparams.brute_short | bool | Generate every 1-, 2-, and 3-letter [a-z] combination and add to the wordlist. Costs ~18,278 extra requests per host; opt-in for thorough scans. | False | -| modules.paramminer_getparams.mutate_case | bool | Also test case-mutated variants of each entry (camelCase for snake_case/kebab-case, Title case for single words). Skipped on URLs with case-insensitive backend extensions like .aspx/.cfm. | False | -| modules.paramminer_getparams.recycle_words | bool | Attempt to use words found during the scan on all other endpoints | False | -| modules.paramminer_getparams.skip_boring_words | bool | Remove commonly uninteresting words from the wordlist | True | -| modules.paramminer_getparams.wordlist | str | Define the wordlist to be used to derive headers. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | -| modules.paramminer_headers.recycle_words | bool | Attempt to use words found during the scan on all other endpoints | False | -| modules.paramminer_headers.skip_boring_words | bool | Remove commonly uninteresting words from the wordlist | True | -| modules.paramminer_headers.wordlist | str | Define the wordlist to be used to derive headers. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | -| modules.portscan.adapter | str | Manually specify a network interface, such as "eth0" or "tun0". If not specified, the first network interface found with a default gateway will be used. | | -| modules.portscan.adapter_ip | str | Send packets using this IP address. Not needed unless masscan's autodetection fails | | -| modules.portscan.adapter_mac | str | Send packets using this as the source MAC address. Not needed unless masscan's autodetection fails | | -| modules.portscan.module_timeout | int | Max time in seconds to spend handling each batch of events | 259200 | -| modules.portscan.ping_first | bool | Only portscan hosts that reply to pings | False | -| modules.portscan.ping_only | bool | Ping sweep only, no portscan | False | -| modules.portscan.ports | str | Ports to scan | | -| modules.portscan.rate | int | Rate in packets per second | 300 | -| modules.portscan.router_mac | str | Send packets to this MAC address as the destination. Not needed unless masscan's autodetection fails | | -| modules.portscan.top_ports | int | Top ports to scan (default 100) (to override, specify 'ports') | 100 | -| modules.portscan.wait | int | Seconds to wait for replies after scan is complete | 5 | -| modules.retirejs.node_version | str | Node.js version to install locally | 18.19.1 | -| modules.retirejs.severity | str | Minimum severity level to report (none, low, medium, high, critical) | medium | -| modules.retirejs.version | str | retire.js version | 5.3.0 | -| modules.robots.include_allow | bool | Include 'Allow' Entries | True | -| modules.robots.include_disallow | bool | Include 'Disallow' Entries | True | -| modules.robots.include_sitemap | bool | Include 'sitemap' entries | False | -| modules.securitytxt.emails | bool | emit EMAIL_ADDRESS events | True | -| modules.securitytxt.urls | bool | emit URL_UNVERIFIED events | True | -| modules.telerik.exploit_RAU_crypto | bool | Attempt to confirm any RAU AXD detections are vulnerable | False | -| modules.telerik.include_subdirs | bool | Include subdirectories in the scan (off by default) | False | -| modules.url_manipulation.allow_redirects | bool | Allowing redirects will sometimes create false positives. Disallowing will sometimes create false negatives. Allowed by default. | True | -| modules.wafw00f.generic_detect | bool | When no specific WAF detections are made, try to perform a generic detect | True | -| modules.webbrute.concurrency | int | Number of concurrent requests per URL being fuzzed | 50 | -| modules.webbrute.extensions | str | Optionally include a list of extensions to extend the keyword with (comma separated or YAML list) | | -| modules.webbrute.ignore_case | bool | Only put lowercase words into the wordlist | False | -| modules.webbrute.lines | int | take only the first N lines from the wordlist when finding directories | 5000 | -| modules.webbrute.max_depth | int | the maximum directory depth to attempt to solve | 0 | -| modules.webbrute.rate | int | Maximum requests per second (0 = unlimited) | 0 | -| modules.webbrute.wordlist | str | Specify wordlist to use when finding directories. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/raft-small-directories.txt | -| modules.webbrute_shortnames.extensions | str | Optionally include a list of extensions to extend the keyword with (comma separated) | | -| modules.webbrute_shortnames.find_common_prefixes | bool | Attempt to automatically detect common prefixes and make additional runs against them | False | -| modules.webbrute_shortnames.find_delimiters | bool | Attempt to detect common delimiters and make additional runs against them | True | -| modules.webbrute_shortnames.find_subwords | bool | Attempt to detect subwords and make additional runs against them | False | -| modules.webbrute_shortnames.max_depth | int | the maximum directory depth to attempt to solve | 1 | -| modules.webbrute_shortnames.max_predictions | int | The maximum number of predictions to generate per shortname prefix | 250 | -| modules.webbrute_shortnames.rate | int | Rate of requests per second (default: 0) | 0 | -| modules.webbrute_shortnames.wordlist_extensions | str | Specify wordlist to use when making extension lists. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | -| modules.wpscan.api_key | str | WPScan API Key | | -| modules.wpscan.connection_timeout | int | The connection timeout in seconds (default 2) | 2 | -| modules.wpscan.disable_tls_checks | bool | Disables the SSL/TLS certificate verification (Default True) | True | -| modules.wpscan.enumerate | str | Enumeration Process see wpscan help documentation (default: vp,vt,cb,dbe) | vp,vt,cb,dbe | -| modules.wpscan.force | bool | Do not check if the target is running WordPress or returns a 403 | False | -| modules.wpscan.request_timeout | int | The request timeout in seconds (default 5) | 5 | -| modules.wpscan.threads | int | How many wpscan threads to spawn (default is 5) | 5 | -| modules.anubisdb.limit | int | Limit the number of subdomains returned per query (increasing this may slow the scan due to garbage results from this API) | 1000 | -| modules.apkpure.output_folder | str | Folder to download APKs to. If not specified, downloaded APKs will be deleted when the scan completes, to minimize disk usage. | | -| modules.bevigil.api_key | str | BeVigil OSINT API Key | | -| modules.bevigil.urls | bool | Emit URLs in addition to DNS_NAMEs | False | -| modules.bucket_file_enum.file_limit | int | Limit the number of files downloaded per bucket | 50 | -| modules.bufferoverrun.api_key | str | BufferOverrun API key | | -| modules.bufferoverrun.commercial | bool | Use commercial API | False | -| modules.builtwith.api_key | str | Builtwith API key | | -| modules.builtwith.redirects | bool | Also look up inbound and outbound redirects | True | -| modules.c99.api_key | str | c99.nl API key | | -| modules.censys_dns.api_key | str | Censys.io API Key in the format of 'key:secret' | | -| modules.censys_dns.max_pages | int | Maximum number of pages to fetch (100 results per page) | 5 | -| modules.censys_ip.api_key | str | Censys.io API Key in the format of 'key:secret' | | -| modules.censys_ip.dns_names_limit | int | Maximum number of DNS names to extract from dns.names (default 100) | 100 | -| modules.censys_ip.in_scope_only | bool | Only query in-scope IPs. If False, will query up to distance 1. | True | -| modules.chaos.api_key | str | Chaos API key | | -| modules.credshed.credshed_url | str | URL of credshed server | | -| modules.credshed.password | str | Credshed password | | -| modules.credshed.username | str | Credshed username | | -| modules.dehashed.api_key | str | DeHashed API Key | | -| modules.dnsbimi.emit_raw_dns_records | bool | Emit RAW_DNS_RECORD events | False | -| modules.dnsbimi.emit_urls | bool | Emit URL_UNVERIFIED events | True | -| modules.dnsbimi.selectors | str | CSV list of BIMI selectors to check | default,email,mail,bimi | -| modules.dnscaa.dns_names | bool | emit DNS_NAME events | True | -| modules.dnscaa.emails | bool | emit EMAIL_ADDRESS events | True | -| modules.dnscaa.in_scope_only | bool | Only check in-scope domains | True | -| modules.dnscaa.urls | bool | emit URL_UNVERIFIED events | True | -| modules.dnstlsrpt.emit_emails | bool | Emit EMAIL_ADDRESS events | True | -| modules.dnstlsrpt.emit_raw_dns_records | bool | Emit RAW_DNS_RECORD events | False | -| modules.dnstlsrpt.emit_urls | bool | Emit URL_UNVERIFIED events | True | -| modules.docker_pull.all_tags | bool | Download all tags from each registry (Default False) | False | -| modules.docker_pull.output_folder | str | Folder to download docker repositories to. If not specified, downloaded docker images will be deleted when the scan completes, to minimize disk usage. | | -| modules.fullhunt.api_key | str | FullHunt API Key | | -| modules.git_clone.api_key | str | Github token | | -| modules.git_clone.output_folder | str | Folder to clone repositories to. If not specified, cloned repositories will be deleted when the scan completes, to minimize disk usage. | | -| modules.gitdumper.fuzz_tags | bool | Fuzz for common git tag names (v0.0.1, 0.0.2, etc.) up to the max_semanic_version | False | -| modules.gitdumper.max_semanic_version | int |` Maximum version number to fuzz for (default < v10.10.10) `| 10 | -| modules.gitdumper.output_folder | str | Folder to download repositories to. If not specified, downloaded repositories will be deleted when the scan completes, to minimize disk usage. | | -| modules.github_codesearch.api_key | str | Github token | | -| modules.github_codesearch.limit | int | Limit code search to this many results | 100 | -| modules.github_org.api_key | str | Github token | | -| modules.github_org.include_member_repos | bool | Also enumerate organization members' repositories | False | -| modules.github_org.include_members | bool | Enumerate organization members | True | -| modules.github_usersearch.api_key | str | Github token | | -| modules.github_workflows.api_key | str | Github token | | -| modules.github_workflows.num_logs | int | For each workflow fetch the last N successful runs logs (max 100) | 1 | -| modules.github_workflows.output_folder | str | Folder to download workflow logs and artifacts to | | -| modules.hunterio.api_key | str | Hunter.IO API key | | -| modules.ip2location.api_key | str | IP2location.io API Key | | -| modules.ip2location.lang | str | Translation information(ISO639-1). The translation is only applicable for continent, country, region and city name. | | -| modules.ipneighbor.num_bits | int | Netmask size (in CIDR notation) to check. Default is 4 bits (16 hosts) | 4 | -| modules.ipstack.api_key | str | IPStack GeoIP API Key | | -| modules.jadx.threads | int | Maximum jadx threads for extracting apk's, default: 4 | 4 | -| modules.kreuzberg.extensions | list | File extensions to parse | ['bak', 'bash', 'bashrc', 'conf', 'cfg', 'crt', 'csv', 'db', 'sqlite', 'doc', 'docx', 'ica', 'indd', 'ini', 'json', 'key', 'pub', 'log', 'markdown', 'md', 'odg', 'odp', 'ods', 'odt', 'pdf', 'pem', 'pps', 'ppsx', 'ppt', 'pptx', 'ps1', 'rdp', 'rsa', 'sh', 'sql', 'swp', 'sxw', 'txt', 'vbs', 'wpd', 'xls', 'xlsx', 'xml', 'yml', 'yaml'] | -| modules.leakix.api_key | str | LeakIX API Key | | -| modules.otx.api_key | str | OTX API key | | -| modules.pgp.search_urls | list | PGP key servers to search |` ['https://keyserver.ubuntu.com/pks/lookup?fingerprint=on&op=vindex&search=<query>', 'http://the.earth.li:11371/pks/lookup?fingerprint=on&op=vindex&search=<query>', 'https://pgpkeys.eu/pks/lookup?search=<query>&op=index', 'https://pgp.mit.edu/pks/lookup?search=<query>&op=index'] `| -| modules.portfilter.allowed_cdn_ports | str | Comma-separated list of ports that are allowed to be scanned for CDNs | 80,443 | -| modules.portfilter.cdn_tags | str | Comma-separated list of tags to skip, e.g. 'cdn,waf' | cdn,waf | -| modules.postman.api_key | str | Postman API Key | | -| modules.postman_download.api_key | str | Postman API Key | | -| modules.postman_download.output_folder | str | Folder to download postman workspaces to. If not specified, downloaded workspaces will be deleted when the scan completes, to minimize disk usage. | | -| modules.securitytrails.api_key | str | SecurityTrails API key | | -| modules.shodan_dns.api_key | str | Shodan API key | | -| modules.shodan_enterprise.api_key | str | Shodan API Key | | -| modules.shodan_enterprise.in_scope_only | bool | Only query in-scope IPs. If False, will query up to distance 1. | True | -| modules.shodan_idb.retries | NoneType | How many times to retry API requests (e.g. after a 429 error). Overrides the global web.api_retries setting. | None | -| modules.subdomainradar.api_key | str | SubDomainRadar.io API key | | -| modules.subdomainradar.group | str | The enumeration group to use. Choose from fast, medium, deep | fast | -| modules.subdomainradar.timeout | int | Timeout in seconds | 120 | -| modules.trajan.ado_token | str | Azure DevOps Personal Access Token (PAT) | | -| modules.trajan.github_token | str | GitHub API token for rate-limiting and private repo access | | -| modules.trajan.gitlab_token | str | GitLab API token for private repo access | | -| modules.trajan.jenkins_password | str | Jenkins password for basic auth | | -| modules.trajan.jenkins_token | str | Jenkins API token | | -| modules.trajan.jenkins_username | str | Jenkins username for basic auth | | -| modules.trajan.jfrog_token | str | JFrog API token | | -| modules.trajan.version | str | Trajan version to download and use | 1.0.0 | -| modules.trickest.api_key | str | Trickest API key | | -| modules.trufflehog.concurrency | int | Number of concurrent workers | 8 | -| modules.trufflehog.config | str | File path or URL to YAML trufflehog config | | -| modules.trufflehog.deleted_forks | bool | Scan for deleted github forks. WARNING: This is SLOW. For a smaller repository, this process can take 20 minutes. For a larger repository, it could take hours. | False | -| modules.trufflehog.only_verified | bool | Only report credentials that have been verified | True | -| modules.trufflehog.version | str | trufflehog version | 3.95.5 | -| modules.urlscan.urls | bool | Emit URLs in addition to DNS_NAMEs | False | -| modules.virustotal.api_key | str | VirusTotal API Key | | -| modules.wayback.archive | bool | fetch archived versions of dead URLs from the Wayback Machine and emit HTTP_RESPONSE events (requires urls=true) | False | -| modules.wayback.garbage_threshold | int | Dedupe similar urls if they are in a group of this size or higher (lower values == less garbage data) | 10 | -| modules.wayback.parameters | bool | emit WEB_PARAMETER events for query parameters discovered in archived URLs (requires urls=true) | False | -| modules.wayback.urls | bool | emit URLs in addition to DNS_NAMEs | False | -| modules.asset_inventory.output_file | str | Set a custom output file | | -| modules.asset_inventory.recheck | bool | When use_previous=True, don't retain past details like open ports or findings. Instead, allow them to be rediscovered by the new scan | False | -| modules.asset_inventory.summary_netmask | int | Subnet mask to use when summarizing IP addresses at end of scan | 16 | -| modules.asset_inventory.use_previous | bool |` Emit previous asset inventory as new events (use in conjunction with -n <old_scan_name>) `| False | -| modules.csv.output_file | str | Output to CSV file | | -| modules.discord.event_types | list | Types of events to send | ['FINDING'] | -| modules.discord.min_severity | str | Only allow FINDING events of this severity or higher | LOW | -| modules.discord.retries | int | Number of times to retry sending the message before skipping the event | 10 | -| modules.discord.webhook_url | str | Discord webhook URL | | -| modules.elastic.password | str | Elastic password | bbotislife | -| modules.elastic.timeout | int | HTTP timeout | 10 | -| modules.elastic.url | str |` Elastic URL (e.g. https://localhost:9200/<your_index>/_doc) `| https://localhost:9200/bbot_events/_doc | -| modules.elastic.username | str | Elastic username | elastic | -| modules.emails.output_file | str | Output to file | | -| modules.json.output_file | str | Output to file | | -| modules.kafka.bootstrap_servers | str | A comma-separated list of Kafka server addresses | localhost:9092 | -| modules.kafka.topic | str | The Kafka topic to publish events to | bbot_events | -| modules.mongo.collection_prefix | str | Prefix the name of each collection with this string | | -| modules.mongo.database | str | The name of the database to use | bbot | -| modules.mongo.password | str | The password to use to connect to the database | | -| modules.mongo.uri | str | The URI of the MongoDB server | mongodb://localhost:27017 | -| modules.mongo.username | str | The username to use to connect to the database | | -| modules.mysql.database | str | The database name to connect to | bbot | -| modules.mysql.host | str | The server running MySQL | localhost | -| modules.mysql.password | str | The password to connect to MySQL | bbotislife | -| modules.mysql.port | int | The port to connect to MySQL | 3306 | -| modules.mysql.retries | int | Number of times to retry connecting to the database (1 second between retries) | 10 | -| modules.mysql.username | str | The username to connect to MySQL | root | -| modules.nats.servers | list | A list of NATS server addresses | [] | -| modules.nats.subject | str | The NATS subject to publish events to | bbot_events | -| modules.neo4j.password | str | Neo4j password | bbotislife | -| modules.neo4j.uri | str | Neo4j server + port | bolt://localhost:7687 | -| modules.neo4j.username | str | Neo4j username | neo4j | -| modules.postgres.database | str | The database name to connect to | bbot | -| modules.postgres.host | str | The server running Postgres | localhost | -| modules.postgres.password | str | The password to connect to Postgres | bbotislife | -| modules.postgres.port | int | The port to connect to Postgres | 5432 | -| modules.postgres.retries | int | Number of times to retry connecting to the database (1 second between retries) | 10 | -| modules.postgres.username | str | The username to connect to Postgres | postgres | -| modules.rabbitmq.queue | str | The RabbitMQ queue to publish events to | bbot_events | -| modules.rabbitmq.url | str | The RabbitMQ connection URL | amqp://guest:guest@localhost/ | -| modules.slack.event_types | list | Types of events to send | ['FINDING'] | -| modules.slack.min_severity | str | Only allow FINDING events of this severity or higher | LOW | -| modules.slack.retries | int | Number of times to retry sending the message before skipping the event | 10 | -| modules.slack.webhook_url | str | Slack webhook URL | | -| modules.splunk.hectoken | str | HEC Token | | -| modules.splunk.index | str | Index to send data to | | -| modules.splunk.source | str | Source path to be added to the metadata | | -| modules.splunk.timeout | int | HTTP timeout | 10 | -| modules.splunk.url | str | Web URL | | -| modules.sqlite.database | str | The path to the sqlite database file | | -| modules.sqlite.retries | int | Number of times to retry connecting to the database (1 second between retries) | 10 | -| modules.stdout.accept_dupes | bool | Whether to show duplicate events, default True | True | -| modules.stdout.event_fields | list | Which event fields to display | [] | -| modules.stdout.event_types | list | Which events to display, default all event types | [] | -| modules.stdout.format | str | Which text format to display, choices: text,json | text | -| modules.stdout.in_scope_only | bool | Whether to only show in-scope events | False | -| modules.subdomains.include_unresolved | bool | Include unresolved subdomains in output | False | -| modules.subdomains.output_file | str | Output to file | | -| modules.teams.event_types | list | Types of events to send | ['FINDING'] | -| modules.teams.min_severity | str | Only allow FINDING events of this severity or higher | LOW | -| modules.teams.retries | int | Number of times to retry sending the message before skipping the event | 10 | -| modules.teams.webhook_url | str | Teams webhook URL | | -| modules.txt.output_file | str | Output to file | | -| modules.web_parameters.include_count | bool | Include the count of each parameter in the output | False | -| modules.web_parameters.output_file | str | Output to file | | -| modules.web_report.css_theme_file | str | CSS theme URL for HTML output | https://cdnjs.cloudflare.com/ajax/libs/github-markdown-css/5.1.0/github-markdown.min.css | -| modules.web_report.output_file | str | Output to file | | -| modules.webhook.bearer | str | Authorization Bearer token | | -| modules.webhook.headers | dict | Additional headers to send with the request | {} | -| modules.webhook.method | str | HTTP method | POST | -| modules.webhook.password | str | Password (basic auth) | | -| modules.webhook.timeout | int | HTTP timeout | 10 | -| modules.webhook.url | str | Web URL | | -| modules.webhook.username | str | Username (basic auth) | | -| modules.websocket.ignore_ssl | bool | Ignores all Websocket SSL related errors (like Self-Signed Certificates, etc.) | False | -| modules.websocket.preserve_graph | bool | Preserve full chains of events in the graph (prevents orphans) | True | -| modules.websocket.token | str | Authorization Bearer token | | -| modules.websocket.url | str | Web URL | | -| modules.zeromq.zmq_address | str | The ZeroMQ socket address to publish events to (e.g. tcp://localhost:5555) | | -| modules.excavate.custom_yara_rules | str | Include custom Yara rules | | -| modules.excavate.speculate_params | bool | Enable speculative parameter extraction from JSON and XML content | False | -| modules.excavate.yara_max_match_data | int | Sets the maximum amount of text that can extracted from a YARA regex | 2000 | -| modules.speculate.essential_only | bool | Only enable essential speculate features (no extra discovery) | False | -| modules.speculate.ip_range_max_hosts | int | Max number of hosts an IP_RANGE can contain to allow conversion into IP_ADDRESS events | 65536 | -| modules.speculate.ports | str | The set of ports to speculate on | 80,443 | +| Config Option | Type | Description | Default | +|--------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| modules.baddns.custom_nameservers | list[str] | Force BadDNS to use a list of custom nameservers | [] | +| modules.baddns.enabled_submodules | list[str] | A list of submodules to enable. Empty list (default) enables CNAME, TXT and MX Only | [] | +| modules.baddns.min_confidence | ConfidenceLiteral | Minimum confidence to emit | MEDIUM | +| modules.baddns.min_severity | SeverityLiteral | Minimum severity to emit | LOW | +| modules.baddns_direct.custom_nameservers | list | Force BadDNS to use a list of custom nameservers | [] | +| modules.baddns_direct.min_confidence | ConfidenceLiteral | Minimum confidence to emit (UNKNOWN, LOW, MEDIUM, HIGH, CONFIRMED) | MEDIUM | +| modules.baddns_direct.min_severity | SeverityLiteral | Minimum severity to emit (INFO, LOW, MEDIUM, HIGH, CRITICAL) | LOW | +| modules.baddns_zone.custom_nameservers | list | Force BadDNS to use a list of custom nameservers | [] | +| modules.baddns_zone.min_confidence | ConfidenceLiteral | Minimum confidence to emit (UNKNOWN, LOW, MEDIUM, HIGH, CONFIRMED) | MEDIUM | +| modules.baddns_zone.min_severity | SeverityLiteral | Minimum severity to emit (INFO, LOW, MEDIUM, HIGH, CRITICAL) | INFO | +| modules.badsecrets.custom_secrets | Optional[str] | Include custom secrets loaded from a local file | None | +| modules.bucket_amazon.permutations | bool | Whether to try permutations | False | +| modules.bucket_digitalocean.permutations | bool | Whether to try permutations | False | +| modules.bucket_firebase.permutations | bool | Whether to try permutations | False | +| modules.bucket_google.permutations | bool | Whether to try permutations | False | +| modules.bucket_hetzner.permutations | bool | Whether to try permutations | False | +| modules.bucket_microsoft.permutations | bool | Whether to try permutations | False | +| modules.dnsbrute.max_depth | int | How many subdomains deep to brute force, i.e. 5.4.3.2.1.evilcorp.com | 5 | +| modules.dnsbrute.recursive_mutations | bool | If True, brute-force hosts discovered by dnsbrute_mutations. The default (False) skips them because the static wordlist heavily overlaps with the mutation algorithm's own output. | False | +| modules.dnsbrute.wordlist | Union[str, list[str]] | Subdomain wordlist URL or file path. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/subdomains-top1million-5000.txt | +| modules.dnsbrute_mutations.max_mutations | int | Maximum number of target-specific mutations to try per subdomain | 100 | +| modules.dnscommonsrv.max_depth | int | The maximum subdomain depth to brute-force SRV records | 2 | +| modules.dnscommonsrv.recursive_mutations | bool | If True, brute-force SRV records on hosts discovered by dnsbrute_mutations. Default False skips them. | False | +| modules.filedownload.extensions | list[str] | File extensions to download | ['bak', 'bash', 'bashrc', 'cfg', 'conf', 'crt', 'csv', 'db', 'dll', 'doc', 'docx', 'exe', 'ica', 'indd', 'ini', 'jar', 'json', 'key', 'log', 'markdown', 'md', 'msi', 'odg', 'odp', 'ods', 'odt', 'pdf', 'pem', 'pps', 'ppsx', 'ppt', 'pptx', 'ps1', 'pub', 'raw', 'rdp', 'rsa', 'sh', 'sql', 'sqlite', 'swp', 'sxw', 'tar.gz', 'tgz', 'tar', 'txt', 'vbs', 'war', 'wpd', 'xls', 'xlsx', 'xml', 'yaml', 'yml', 'zip', 'lzma', 'rar', '7z', 'xz', 'bz2'] | +| modules.filedownload.max_filesize | str | Cancel download if filesize is greater than this size | 10MB | +| modules.filedownload.output_folder | str | Folder to download files to. If not specified, downloaded files will be deleted when the scan completes, to minimize disk usage. | | +| modules.fingerprintx.skip_common_web | bool | Skip common web ports such as 80, 443, 8080, 8443, etc. | True | +| modules.fingerprintx.version | str | fingerprintx version | 1.1.4 | +| modules.generic_ssrf.skip_dns_interaction | bool | Do not report DNS interactions (only HTTP interaction) | False | +| modules.gitlab_com.api_key | str | list[str] | GitLab access token (for gitlab.com/org only) | | +| modules.gitlab_onprem.api_key | str | list[str] | GitLab access token (for self-hosted instances only) | | +| modules.gowitness.chrome_path | str | Path to chrome executable | | +| modules.gowitness.idle_timeout | int | Skip the current gowitness batch if it stalls for longer than this many seconds | 1800 | +| modules.gowitness.output_path | str | Where to save screenshots | | +| modules.gowitness.resolution_x | int | Screenshot resolution x | 1440 | +| modules.gowitness.resolution_y | int | Screenshot resolution y | 900 | +| modules.gowitness.social | bool | Whether to screenshot social media webpages | False | +| modules.gowitness.threads | int | How many gowitness threads to spawn (default is number of CPUs x 2) | 0 | +| modules.gowitness.timeout | int | Preflight check timeout | 10 | +| modules.gowitness.version | str | Gowitness version | 3.1.1 | +| modules.graphql_introspection.graphql_endpoint_urls | list[str] | List of GraphQL endpoint to suffix to the target URL | ['/', '/graphql', '/v1/graphql'] | +| modules.graphql_introspection.output_folder | str | Folder to save the GraphQL schemas to | | +| modules.http.in_scope_only | bool | Only visit web resources that are in scope. | True | +| modules.http.max_response_size | int | Max response size in bytes | 5242880 | +| modules.http.store_responses | bool | Save raw HTTP responses to scan folder | False | +| modules.http.threads | int | Number of concurrent requests | 50 | +| modules.iis_shortnames.detect_only | bool | Only detect the vulnerability and do not run the shortname scanner | True | +| modules.iis_shortnames.max_node_count | int | Limit how many nodes to attempt to resolve on any given recursion branch | 50 | +| modules.iis_shortnames.speculate_magic_urls | bool | Attempt to discover iis 'magic' special folders | True | +| modules.legba.concurrency | int | Number of concurrent workers, gets overridden for SSH | 3 | +| modules.legba.ftp_wordlist | Union[str, list[str]] | Wordlist for FTP combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/ftp-betterdefaultpasslist.txt | +| modules.legba.mssql_wordlist | Union[str, list[str]] | Wordlist for MSSQL combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/mssql-betterdefaultpasslist.txt | +| modules.legba.mysql_wordlist | Union[str, list[str]] | Wordlist for MySQL combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/mysql-betterdefaultpasslist.txt | +| modules.legba.postgresql_wordlist | Union[str, list[str]] | Wordlist for PostgreSQL combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/postgres-betterdefaultpasslist.txt | +| modules.legba.rate_limit | int | Limit the number of requests per second, gets overridden for SSH | 3 | +| modules.legba.ssh_wordlist | Union[str, list[str]] | Wordlist for SSH combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/ssh-betterdefaultpasslist.txt | +| modules.legba.telnet_wordlist | Union[str, list[str]] | Wordlist for TELNET combined username:password, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/telnet-betterdefaultpasslist.txt | +| modules.legba.version | str | legba version | 1.1.1 | +| modules.legba.vnc_wordlist | Union[str, list[str]] | Wordlist for VNC passwords, newline separated. Accepts a URL or local file path, or a list of URLs/paths to merge multiple wordlists (duplicates removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Passwords/Default-Credentials/vnc-betterdefaultpasslist.txt | +| modules.lightfuzz.avoid_wafs | bool | Avoid running against confirmed WAFs, which are likely to block lightfuzz requests | True | +| modules.lightfuzz.disable_post | bool | Disable processing of POST parameters, avoiding form submissions. | False | +| modules.lightfuzz.emit_baseline_responses | bool | Emit canonical baseline responses as HTTP_RESPONSE events so excavate can mine them for new params/URLs. | True | +| modules.lightfuzz.enabled_submodules | list[str] | A list of submodules to enable. Empty list enabled all modules. | ['sqli', 'cmdi', 'xss', 'path', 'ssti', 'crypto', 'serial', 'esi', 'ssrf'] | +| modules.lightfuzz.force_common_headers | bool | Force emit commonly exploitable parameters that may be difficult to detect | False | +| modules.lightfuzz.try_get_as_post | bool | For each GETPARAM, also fuzz it as a POSTPARAM (in addition to normal GET fuzzing). | False | +| modules.lightfuzz.try_post_as_get | bool | For each POSTPARAM, also fuzz it as a GETPARAM (in addition to normal POST fuzzing). | False | +| modules.medusa.snmp_versions | list[str] | List of SNMP versions to attempt against the SNMP server (default ['1', '2C']) | ['1', '2C'] | +| modules.medusa.snmp_wordlist | Union[str, list[str]] | Wordlist url for SNMP community strings, newline separated (default https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Discovery/SNMP/snmp.txt). Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Discovery/SNMP/common-snmp-community-strings.txt | +| modules.medusa.threads | int | Number of communities to be tested concurrently (default 5) | 5 | +| modules.medusa.timeout_s | int | Wait time for the SNMP response(s) once at the end of all attempts (default 5) | 5 | +| modules.medusa.wait_microseconds | int | Wait time after every SNMP request in microseconds (default 200) | 200 | +| modules.ntlm.try_all | bool | Try every NTLM endpoint | False | +| modules.nuclei.batch_size | int | Number of targets to send to Nuclei per batch (default 200) | 200 | +| modules.nuclei.budget | int | Used in budget mode to set the number of allowed requests per host | 1 | +| modules.nuclei.concurrency | int | maximum number of templates to be executed in parallel (default 25) | 25 | +| modules.nuclei.directory_only | bool | Filter out 'file' URL event (default True) | True | +| modules.nuclei.etags | str | tags to exclude from the scan | | +| modules.nuclei.mode | Literal['manual', 'technology', 'severe', 'budget'] | manual | technology | severe | budget. Technology: Only activate based on technology events that match nuclei tags (nuclei -as mode). Manual (DEFAULT): Fully manual settings. Severe: Only critical and high severity templates without intrusive. Budget: Limit Nuclei to a specified number of HTTP requests | manual | +| modules.nuclei.module_timeout | int | Max time in seconds to spend handling each batch of events | 21600 | +| modules.nuclei.ratelimit | int | maximum number of requests to send per second (default 150) | 150 | +| modules.nuclei.retries | int | number of times to retry a failed request (default 0) | 0 | +| modules.nuclei.severity | str | Filter based on severity field available in the template. | | +| modules.nuclei.silent | bool | Don't display nuclei's banner or status messages | False | +| modules.nuclei.tags | str | execute a subset of templates that contain the provided tags | | +| modules.nuclei.templates | str | template or template directory paths to include in the scan | | +| modules.nuclei.version | str | nuclei version | 3.9.0 | +| modules.oauth.try_all | bool | Check for OAUTH/IODC on every subdomain and URL. | False | +| modules.paramminer_cookies.recycle_words | bool | Attempt to use words found during the scan on all other endpoints | False | +| modules.paramminer_cookies.skip_boring_words | bool | Remove commonly uninteresting words from the wordlist | True | +| modules.paramminer_cookies.wordlist | Union[str, list[str]] | Define the wordlist to be used to derive cookies. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | +| modules.paramminer_getparams.brute_short | bool | Generate every 1-, 2-, and 3-letter [a-z] combination and add to the wordlist. Costs ~18,278 extra requests per host; opt-in for thorough scans. | False | +| modules.paramminer_getparams.mutate_case | bool | Also test case-mutated variants of each entry (camelCase for snake_case/kebab-case, Title case for single words). Skipped on URLs with case-insensitive backend extensions like .aspx/.cfm. | False | +| modules.paramminer_getparams.recycle_words | bool | Attempt to use words found during the scan on all other endpoints | False | +| modules.paramminer_getparams.skip_boring_words | bool | Remove commonly uninteresting words from the wordlist | True | +| modules.paramminer_getparams.wordlist | Union[str, list[str]] | Define the wordlist to be used to derive headers. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | +| modules.paramminer_headers.recycle_words | bool | Attempt to use words found during the scan on all other endpoints | False | +| modules.paramminer_headers.skip_boring_words | bool | Remove commonly uninteresting words from the wordlist | True | +| modules.paramminer_headers.wordlist | Union[str, list[str]] | Define the wordlist to be used to derive headers. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | +| modules.portscan.adapter | str | Manually specify a network interface, such as "eth0" or "tun0". If not specified, the first network interface found with a default gateway will be used. | | +| modules.portscan.adapter_ip | str | Send packets using this IP address. Not needed unless masscan's autodetection fails | | +| modules.portscan.adapter_mac | str | Send packets using this as the source MAC address. Not needed unless masscan's autodetection fails | | +| modules.portscan.module_timeout | int | Max time in seconds to spend handling each batch of events | 259200 | +| modules.portscan.ping_first | bool | Only portscan hosts that reply to pings | False | +| modules.portscan.ping_only | bool | Ping sweep only, no portscan | False | +| modules.portscan.ports | str | Ports to scan | | +| modules.portscan.rate | int | Rate in packets per second | 300 | +| modules.portscan.router_mac | str | Send packets to this MAC address as the destination. Not needed unless masscan's autodetection fails | | +| modules.portscan.skip_tags | str | Comma-separated event tags that will be excluded from scanning (e.g. 'cdn,waf'). speculate will emit assumed-open ports for these instead. | | +| modules.portscan.top_ports | int | Top ports to scan (default 100) (to override, specify 'ports') | 100 | +| modules.portscan.wait | int | Seconds to wait for replies after scan is complete | 5 | +| modules.retirejs.node_version | str | Node.js version to install locally | 18.19.1 | +| modules.retirejs.severity | Literal['none', 'low', 'medium', 'high', 'critical'] | Minimum severity level to report (none, low, medium, high, critical) | medium | +| modules.retirejs.version | str | retire.js version | 5.3.0 | +| modules.robots.include_allow | bool | Include 'Allow' Entries | True | +| modules.robots.include_disallow | bool | Include 'Disallow' Entries | True | +| modules.robots.include_sitemap | bool | Include 'sitemap' entries | False | +| modules.securitytxt.emails | bool | emit EMAIL_ADDRESS events | True | +| modules.securitytxt.urls | bool | emit URL_UNVERIFIED events | True | +| modules.telerik.exploit_RAU_crypto | bool | Attempt to confirm any RAU AXD detections are vulnerable | False | +| modules.telerik.include_subdirs | bool | Include subdirectories in the scan (off by default) | False | +| modules.url_manipulation.allow_redirects | bool | Allowing redirects will sometimes create false positives. Disallowing will sometimes create false negatives. Allowed by default. | True | +| modules.virtualhost.brute_lines | int | Take only the first N lines from the wordlist when finding directories | 2000 | +| modules.virtualhost.brute_wordlist | str | Wordlist containing subdomains | https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/subdomains-top1million-5000.txt | +| modules.virtualhost.certificate_sans | bool | Enable extraction and testing of Subject Alternative Names from certificates | False | +| modules.virtualhost.force_basehost | str | Use a custom base host (e.g. evilcorp.com) instead of the default behavior of using the current URL | | +| modules.virtualhost.max_concurrent_requests | int | Maximum number of concurrent virtual host requests | 80 | +| modules.virtualhost.mutation_check | bool | Enable trying mutations of the target host | True | +| modules.virtualhost.report_interesting_default_content | bool | Report interesting default content | True | +| modules.virtualhost.require_inaccessible | bool | Only test virtual hosts that are not directly accessible (for discovering hidden content) | True | +| modules.virtualhost.special_hosts | bool | Enable testing of special virtual host list (localhost, etc.) | False | +| modules.virtualhost.subdomain_brute | bool | Enable subdomain brute-force on target host | True | +| modules.virtualhost.wordcloud_check | bool | Enable check using scan-wide wordcloud data on target host | False | +| modules.waf_bypass.neighbor_cidr | int | CIDR mask (24-31) used for neighbor enumeration when search_ip_neighbors is true | 24 | +| modules.waf_bypass.search_ip_neighbors | bool | Also check IP neighbors of the target domain | True | +| modules.waf_bypass.similarity_threshold | float | Similarity threshold for content matching | 0.9 | +| modules.wafw00f.generic_detect | bool | When no specific WAF detections are made, try to perform a generic detect | True | +| modules.webbrute.avoid_wafs | bool | Avoid running against confirmed WAFs, which are likely to block brute-force requests | True | +| modules.webbrute.concurrency | int | Number of concurrent requests per URL being fuzzed | 50 | +| modules.webbrute.extensions | Union[str, list[str]] | Optionally include a list of extensions to extend the keyword with (comma separated or YAML list) | | +| modules.webbrute.ignore_case | bool | Only put lowercase words into the wordlist | False | +| modules.webbrute.lines | int | take only the first N lines from the wordlist when finding directories | 5000 | +| modules.webbrute.max_depth | int | the maximum directory depth to attempt to solve | 0 | +| modules.webbrute.rate | int | Maximum requests per second (0 = unlimited) | 0 | +| modules.webbrute.wordlist | Union[str, list[str]] | Specify wordlist to use when finding directories. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/raft-small-directories.txt | +| modules.webbrute_shortnames.extensions | str | Optionally include a list of extensions to extend the keyword with (comma separated) | | +| modules.webbrute_shortnames.find_common_prefixes | bool | Attempt to automatically detect common prefixes and make additional runs against them | False | +| modules.webbrute_shortnames.find_delimiters | bool | Attempt to detect common delimiters and make additional runs against them | True | +| modules.webbrute_shortnames.find_subwords | bool | Attempt to detect subwords and make additional runs against them | False | +| modules.webbrute_shortnames.max_depth | int | the maximum directory depth to attempt to solve | 1 | +| modules.webbrute_shortnames.max_predictions | int | The maximum number of predictions to generate per shortname prefix | 250 | +| modules.webbrute_shortnames.rate | int | Rate of requests per second (default: 0) | 0 | +| modules.webbrute_shortnames.wordlist_extensions | Union[str, list[str]] | Specify wordlist to use when making extension lists. Accepts a list of URLs/paths to merge multiple wordlists (duplicates are removed). | | +| modules.anubisdb.limit | int | Limit the number of subdomains returned per query (increasing this may slow the scan due to garbage results from this API) | 1000 | +| modules.apkpure.output_folder | str | Folder to download APKs to. If not specified, downloaded APKs will be deleted when the scan completes, to minimize disk usage. | | +| modules.bevigil.api_key | str | list[str] | BeVigil OSINT API Key | | +| modules.bevigil.urls | bool | Emit URLs in addition to DNS_NAMEs | False | +| modules.bucket_file_enum.file_limit | int | Limit the number of files downloaded per bucket | 50 | +| modules.bufferoverrun.api_key | str | list[str] | BufferOverrun API key | | +| modules.bufferoverrun.commercial | bool | Use commercial API | False | +| modules.builtwith.api_key | str | list[str] | Builtwith API key | | +| modules.builtwith.redirects | bool | Also look up inbound and outbound redirects | True | +| modules.c99.api_key | str | list[str] | c99.nl API key | | +| modules.censys_dns.api_key | str | list[str] | Censys.io API Key in the format of 'key:secret' | | +| modules.censys_dns.max_pages | int | Maximum number of pages to fetch (100 results per page) | 5 | +| modules.censys_ip.api_key | str | list[str] | Censys.io API Key in the format of 'key:secret' | | +| modules.censys_ip.dns_names_limit | int | Maximum number of DNS names to extract from dns.names (default 100) | 100 | +| modules.censys_ip.in_scope_only | bool | Only query in-scope IPs. If False, will query up to distance 1. | True | +| modules.certspotter.api_key | str | SSLMate API key (enables pagination and higher rate limits) | | +| modules.chaos.api_key | str | list[str] | Chaos API key | | +| modules.credshed.credshed_url | str | URL of credshed server | | +| modules.credshed.password | str | Credshed password | | +| modules.credshed.username | str | Credshed username | | +| modules.dehashed.api_key | str | list[str] | DeHashed API Key | | +| modules.dnsbimi.emit_raw_dns_records | bool | Emit RAW_DNS_RECORD events | False | +| modules.dnsbimi.emit_urls | bool | Emit URL_UNVERIFIED events | True | +| modules.dnsbimi.selectors | str | CSV list of BIMI selectors to check | default,email,mail,bimi | +| modules.dnscaa.dns_names | bool | emit DNS_NAME events | True | +| modules.dnscaa.emails | bool | emit EMAIL_ADDRESS events | True | +| modules.dnscaa.in_scope_only | bool | Only check in-scope domains | True | +| modules.dnscaa.urls | bool | emit URL_UNVERIFIED events | True | +| modules.dnstlsrpt.emit_emails | bool | Emit EMAIL_ADDRESS events | True | +| modules.dnstlsrpt.emit_raw_dns_records | bool | Emit RAW_DNS_RECORD events | False | +| modules.dnstlsrpt.emit_urls | bool | Emit URL_UNVERIFIED events | True | +| modules.docker_pull.all_tags | bool | Download all tags from each registry (Default False) | False | +| modules.docker_pull.output_folder | str | Folder to download docker repositories to. If not specified, downloaded docker images will be deleted when the scan completes, to minimize disk usage. | | +| modules.fullhunt.api_key | str | list[str] | FullHunt API Key | | +| modules.git_clone.api_key | str | list[str] | Github token | | +| modules.git_clone.output_folder | str | Folder to clone repositories to. If not specified, cloned repositories will be deleted when the scan completes, to minimize disk usage. | | +| modules.gitdumper.fuzz_tags | bool | Fuzz for common git tag names (v0.0.1, 0.0.2, etc.) up to the max_semanic_version | False | +| modules.gitdumper.max_semanic_version | int |` Maximum version number to fuzz for (default < v10.10.10) `| 10 | +| modules.gitdumper.output_folder | str | Folder to download repositories to. If not specified, downloaded repositories will be deleted when the scan completes, to minimize disk usage. | | +| modules.github_codesearch.api_key | str | list[str] | Github token | | +| modules.github_codesearch.limit | int | Limit code search to this many results | 100 | +| modules.github_org.api_key | str | list[str] | Github token | | +| modules.github_org.include_member_repos | bool | Also enumerate organization members' repositories | False | +| modules.github_org.include_members | bool | Enumerate organization members | True | +| modules.github_usersearch.api_key | str | list[str] | Github token | | +| modules.github_workflows.api_key | str | list[str] | Github token | | +| modules.github_workflows.num_logs | int | For each workflow fetch the last N successful runs logs (max 100) | 1 | +| modules.github_workflows.output_folder | str | Folder to download workflow logs and artifacts to | | +| modules.hunterio.api_key | str | list[str] | Hunter.IO API key | | +| modules.ip2location.api_key | str | list[str] | IP2location.io API Key | | +| modules.ip2location.lang | str | Translation information(ISO639-1). The translation is only applicable for continent, country, region and city name. | | +| modules.ipneighbor.num_bits | int | Netmask size (in CIDR notation) to check. Default is 4 bits (16 hosts) | 4 | +| modules.ipstack.api_key | str | list[str] | IPStack GeoIP API Key | | +| modules.jadx.threads | int | Maximum jadx threads for extracting apk's, default: 4 | 4 | +| modules.kreuzberg.extensions | list[str] | File extensions to parse | ['bak', 'bash', 'bashrc', 'conf', 'cfg', 'crt', 'csv', 'db', 'sqlite', 'doc', 'docx', 'ica', 'indd', 'ini', 'json', 'key', 'pub', 'log', 'markdown', 'md', 'odg', 'odp', 'ods', 'odt', 'pdf', 'pem', 'pps', 'ppsx', 'ppt', 'pptx', 'ps1', 'rdp', 'rsa', 'sh', 'sql', 'swp', 'sxw', 'txt', 'vbs', 'wpd', 'xls', 'xlsx', 'xml', 'yml', 'yaml'] | +| modules.leakix.api_key | str | list[str] | LeakIX API Key | | +| modules.otx.api_key | str | list[str] | OTX API key | | +| modules.pgp.search_urls | list[str] | PGP key servers to search |` ['https://keyserver.ubuntu.com/pks/lookup?fingerprint=on&op=vindex&search=<query>', 'http://the.earth.li:11371/pks/lookup?fingerprint=on&op=vindex&search=<query>', 'https://pgpkeys.eu/pks/lookup?search=<query>&op=index', 'https://pgp.mit.edu/pks/lookup?search=<query>&op=index'] `| +| modules.portfilter.allowed_cdn_ports | str | Comma-separated list of ports that are allowed to be scanned for CDNs | 80,443 | +| modules.portfilter.cdn_tags | str | Comma-separated list of tags to skip, e.g. 'cdn,waf' | cdn,waf | +| modules.postman.api_key | str | list[str] | Postman API Key | | +| modules.postman_download.api_key | str | list[str] | Postman API Key | | +| modules.postman_download.output_folder | str | Folder to download postman workspaces to. If not specified, downloaded workspaces will be deleted when the scan completes, to minimize disk usage. | | +| modules.securitytrails.api_key | str | list[str] | SecurityTrails API key | | +| modules.shodan_dns.api_key | str | list[str] | Shodan API key | | +| modules.shodan_enterprise.api_key | str | list[str] | Shodan API Key | | +| modules.shodan_enterprise.in_scope_only | bool | Only query in-scope IPs. If False, will query up to distance 1. | True | +| modules.shodan_idb.retries | Optional[int] | How many times to retry API requests (e.g. after a 429 error). Overrides the global web.api_retries setting. | None | +| modules.subdomainradar.api_key | str | list[str] | SubDomainRadar.io API key | | +| modules.subdomainradar.group | Literal['fast', 'medium', 'deep'] | The enumeration group to use. Choose from fast, medium, deep | fast | +| modules.subdomainradar.timeout | int | Timeout in seconds | 120 | +| modules.trajan.ado_token | str | Azure DevOps Personal Access Token (PAT) | | +| modules.trajan.github_token | str | GitHub API token for rate-limiting and private repo access | | +| modules.trajan.gitlab_token | str | GitLab API token for private repo access | | +| modules.trajan.jenkins_password | str | Jenkins password for basic auth | | +| modules.trajan.jenkins_token | str | Jenkins API token | | +| modules.trajan.jenkins_username | str | Jenkins username for basic auth | | +| modules.trajan.jfrog_token | str | JFrog API token | | +| modules.trajan.version | str | Trajan version to download and use | 1.0.0 | +| modules.trickest.api_key | str | list[str] | Trickest API key | | +| modules.trufflehog.concurrency | int | Number of concurrent workers | 8 | +| modules.trufflehog.config | str | File path or URL to YAML trufflehog config | | +| modules.trufflehog.deleted_forks | bool | Scan for deleted github forks. WARNING: This is SLOW. For a smaller repository, this process can take 20 minutes. For a larger repository, it could take hours. | False | +| modules.trufflehog.only_verified | bool | Only report credentials that have been verified | True | +| modules.trufflehog.version | str | trufflehog version | 3.95.5 | +| modules.urlscan.urls | bool | Emit URLs in addition to DNS_NAMEs | False | +| modules.virustotal.api_key | str | list[str] | VirusTotal API Key | | +| modules.wayback.archive | bool | fetch archived versions of dead URLs from the Wayback Machine and emit HTTP_RESPONSE events (requires urls=true) | False | +| modules.wayback.garbage_threshold | int | Dedupe similar urls if they are in a group of this size or higher (lower values == less garbage data) | 10 | +| modules.wayback.max_records | int | Maximum number of URLs to fetch from the CDX API | 100000 | +| modules.wayback.parameters | bool | emit WEB_PARAMETER events for query parameters discovered in archived URLs (requires urls=true) | False | +| modules.wayback.urls | bool | emit URLs in addition to DNS_NAMEs | False | +| modules.asset_inventory.output_file | str | Set a custom output file | | +| modules.asset_inventory.recheck | bool | When use_previous=True, don't retain past details like open ports or findings. Instead, allow them to be rediscovered by the new scan | False | +| modules.asset_inventory.summary_netmask | int | Subnet mask to use when summarizing IP addresses at end of scan | 16 | +| modules.asset_inventory.use_previous | bool |` Emit previous asset inventory as new events (use in conjunction with -n <old_scan_name>) `| False | +| modules.csv.output_file | str | Output to CSV file | | +| modules.discord.event_types | list[str] | Types of events to send | ['FINDING'] | +| modules.discord.min_severity | str | Only allow FINDING events of this severity or higher | LOW | +| modules.discord.retries | int | Number of times to retry sending the message before skipping the event | 10 | +| modules.discord.webhook_url | str | Discord webhook URL | | +| modules.elastic.password | str | Elastic password | bbotislife | +| modules.elastic.timeout | int | HTTP timeout | 10 | +| modules.elastic.url | str |` Elastic URL (e.g. https://localhost:9200/<your_index>/_doc) `| https://localhost:9200/bbot_events/_doc | +| modules.elastic.username | str | Elastic username | elastic | +| modules.emails.output_file | str | Output to file | | +| modules.json.output_file | str | Output to file | | +| modules.kafka.bootstrap_servers | str | A comma-separated list of Kafka server addresses | localhost:9092 | +| modules.kafka.topic | str | The Kafka topic to publish events to | bbot_events | +| modules.mongo.collection_prefix | str | Prefix the name of each collection with this string | | +| modules.mongo.database | str | The name of the database to use | bbot | +| modules.mongo.password | str | The password to use to connect to the database | | +| modules.mongo.uri | str | The URI of the MongoDB server | mongodb://localhost:27017 | +| modules.mongo.username | str | The username to use to connect to the database | | +| modules.mysql.database | str | The database name to connect to | bbot | +| modules.mysql.host | str | The server running MySQL | localhost | +| modules.mysql.password | str | The password to connect to MySQL | bbotislife | +| modules.mysql.port | int | The port to connect to MySQL | 3306 | +| modules.mysql.retries | int | Number of times to retry connecting to the database (1 second between retries) | 10 | +| modules.mysql.username | str | The username to connect to MySQL | root | +| modules.nats.servers | list | A list of NATS server addresses | [] | +| modules.nats.subject | str | The NATS subject to publish events to | bbot_events | +| modules.neo4j.password | str | Neo4j password | bbotislife | +| modules.neo4j.uri | str | Neo4j server + port | bolt://localhost:7687 | +| modules.neo4j.username | str | Neo4j username | neo4j | +| modules.postgres.database | str | The database name to connect to | bbot | +| modules.postgres.host | str | The server running Postgres | localhost | +| modules.postgres.password | str | The password to connect to Postgres | bbotislife | +| modules.postgres.port | int | The port to connect to Postgres | 5432 | +| modules.postgres.retries | int | Number of times to retry connecting to the database (1 second between retries) | 10 | +| modules.postgres.username | str | The username to connect to Postgres | postgres | +| modules.rabbitmq.queue | str | The RabbitMQ queue to publish events to | bbot_events | +| modules.rabbitmq.url | str | The RabbitMQ connection URL | amqp://guest:guest@localhost/ | +| modules.slack.event_types | list[str] | Types of events to send | ['FINDING'] | +| modules.slack.min_severity | str | Only allow FINDING events of this severity or higher | LOW | +| modules.slack.retries | int | Number of times to retry sending the message before skipping the event | 10 | +| modules.slack.webhook_url | str | Slack webhook URL | | +| modules.splunk.hectoken | str | HEC Token | | +| modules.splunk.index | str | Index to send data to | | +| modules.splunk.source | str | Source path to be added to the metadata | | +| modules.splunk.timeout | int | HTTP timeout | 10 | +| modules.splunk.url | str | Web URL | | +| modules.sqlite.database | str | The path to the sqlite database file | | +| modules.sqlite.retries | int | Number of times to retry connecting to the database (1 second between retries) | 10 | +| modules.stdout.accept_dupes | bool | Whether to show duplicate events, default True | True | +| modules.stdout.event_fields | list | Which event fields to display | [] | +| modules.stdout.event_types | list | Which events to display, default all event types | [] | +| modules.stdout.format | Literal['text', 'json'] | Which text format to display, choices: text,json | text | +| modules.stdout.in_scope_only | bool | Whether to only show in-scope events | False | +| modules.subdomains.include_unresolved | bool | Include unresolved subdomains in output | False | +| modules.subdomains.output_file | str | Output to file | | +| modules.teams.event_types | list[str] | Types of events to send | ['FINDING'] | +| modules.teams.min_severity | str | Only allow FINDING events of this severity or higher | LOW | +| modules.teams.retries | int | Number of times to retry sending the message before skipping the event | 10 | +| modules.teams.webhook_url | str | Teams webhook URL | | +| modules.txt.output_file | str | Output to file | | +| modules.web_parameters.include_count | bool | Include the count of each parameter in the output | False | +| modules.web_parameters.output_file | str | Output to file | | +| modules.web_report.css_theme_file | str | CSS theme URL for HTML output | https://cdnjs.cloudflare.com/ajax/libs/github-markdown-css/5.1.0/github-markdown.min.css | +| modules.web_report.output_file | str | Output to file | | +| modules.webhook.bearer | str | Authorization Bearer token | | +| modules.webhook.headers | dict | Additional headers to send with the request | {} | +| modules.webhook.method | str | HTTP method | POST | +| modules.webhook.password | str | Password (basic auth) | | +| modules.webhook.timeout | int | HTTP timeout | 10 | +| modules.webhook.url | str | Web URL | | +| modules.webhook.username | str | Username (basic auth) | | +| modules.websocket.ignore_ssl | bool | Ignores all Websocket SSL related errors (like Self-Signed Certificates, etc.) | False | +| modules.websocket.preserve_graph | bool | Preserve full chains of events in the graph (prevents orphans) | True | +| modules.websocket.token | str | Authorization Bearer token | | +| modules.websocket.url | str | Web URL | | +| modules.zeromq.zmq_address | str | The ZeroMQ socket address to publish events to (e.g. tcp://localhost:5555) | | +| modules.excavate.custom_yara_rules | str | Include custom Yara rules | | +| modules.excavate.max_form_bytes | int |` Maximum byte slice of the response body searched for a single <form> body. YARA only locates form openings; the bounded slice is what the Python re-based extractor scans for fields. Caps worst-case extraction work per form match. `| 262144 | +| modules.excavate.speculate_params | bool | Enable speculative parameter extraction from JSON and XML content | False | +| modules.excavate.yara_max_match_data | int | Sets the maximum amount of text that can extracted from a YARA regex | 2000 | +| modules.speculate.essential_only | bool | Only enable essential speculate features (no extra discovery) | False | +| modules.speculate.ip_range_max_hosts | int | Max number of hosts an IP_RANGE can contain to allow conversion into IP_ADDRESS events | 65536 | +| modules.speculate.ports | str | The set of ports to speculate on | 80,443 | <!-- END BBOT MODULE OPTIONS --> diff --git a/docs/scanning/events.md b/docs/scanning/events.md index 6f1687b382..efc380ef90 100644 --- a/docs/scanning/events.md +++ b/docs/scanning/events.md @@ -117,40 +117,42 @@ Below is a full list of event types along with which modules produce/consume the ## List of Event Types <!-- BBOT EVENTS --> -| Event Type | # Consuming Modules | # Producing Modules | Consuming Modules | Producing Modules | -|---------------------|-----------------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| * | 24 | 0 | affiliates, cloudcheck, csv, discord, dnsresolve, elastic, json, kafka, mongo, mysql, nats, neo4j, postgres, python, rabbitmq, slack, splunk, sqlite, stdout, teams, txt, webhook, websocket, zeromq | | -| ASN | 0 | 1 | | asn | -| AZURE_TENANT | 1 | 1 | speculate | azure_tenant | -| CODE_REPOSITORY | 8 | 8 | docker_pull, git_clone, gitdumper, github_workflows, google_playstore, postman_download, trajan, trufflehog | code_repository, dockerhub, git, github_codesearch, github_org, gitlab_com, gitlab_onprem, postman | -| DNS_NAME | 57 | 39 | anubisdb, asset_inventory, azure_tenant, baddns, baddns_zone, bevigil, bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bufferoverrun, builtwith, c99, censys_dns, certspotter, chaos, credshed, crt, crt_db, dehashed, dnsbimi, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, emailformat, fullhunt, github_codesearch, github_usersearch, hackertarget, hunterio, leakix, myssl, nmap_xml, oauth, otx, pgp, portscan, rapiddns, securitytrails, securitytxt, shodan_dns, shodan_idb, skymem, speculate, subdomaincenter, subdomainradar, subdomains, trickest, urlscan, viewdns, virustotal, wayback | anubisdb, azure_tenant, bevigil, bufferoverrun, builtwith, c99, censys_dns, censys_ip, certspotter, chaos, crt, crt_db, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnsresolve, fullhunt, hackertarget, hunterio, leakix, myssl, ntlm, oauth, otx, rapiddns, securitytrails, shodan_dns, shodan_idb, speculate, sslcert, subdomaincenter, subdomainradar, trickest, urlscan, viewdns, virustotal, wayback | -| DNS_NAME_UNRESOLVED | 3 | 0 | baddns, speculate, subdomains | | -| EMAIL_ADDRESS | 1 | 11 | emails | credshed, dehashed, dnscaa, dnstlsrpt, emailformat, github_usersearch, hunterio, pgp, securitytxt, skymem, sslcert | -| FILESYSTEM | 4 | 9 | jadx, kreuzberg, trufflehog, unarchive | apkpure, docker_pull, filedownload, git_clone, gitdumper, github_workflows, jadx, postman_download, unarchive | -| FINDING | 2 | 37 | asset_inventory, web_report | ajaxpro, aspnet_bin_exposure, azure_tenant, baddns, baddns_direct, baddns_zone, badsecrets, bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bypass403, dotnetnuke, generic_ssrf, git, gitlab_onprem, graphql_introspection, host_header, hunt, legba, lightfuzz, medusa, newsletters, ntlm, nuclei, reflected_parameters, retirejs, shodan_enterprise, shodan_idb, speculate, telerik, trajan, trufflehog, url_manipulation, wpscan | -| GEOLOCATION | 0 | 2 | | ip2location, ipstack | -| HASHED_PASSWORD | 0 | 2 | | credshed, dehashed | -| HTTP_RESPONSE | 19 | 1 | ajaxpro, asset_inventory, badsecrets, dotnetnuke, excavate, filedownload, gitlab_onprem, host_header, newsletters, nmap_xml, ntlm, paramminer_cookies, paramminer_getparams, paramminer_headers, speculate, sslcert, telerik, trufflehog, wpscan | http | -| IP_ADDRESS | 11 | 5 | asn, asset_inventory, censys_ip, ip2location, ipneighbor, ipstack, nmap_xml, portscan, shodan_enterprise, shodan_idb, speculate | asset_inventory, censys_ip, dnsresolve, ipneighbor, speculate | -| IP_RANGE | 2 | 0 | portscan, speculate | | -| MOBILE_APP | 1 | 1 | apkpure | google_playstore | -| OPEN_TCP_PORT | 5 | 6 | asset_inventory, fingerprintx, http, nmap_xml, portfilter | asset_inventory, censys_ip, portscan, shodan_enterprise, shodan_idb, speculate | -| OPEN_UDP_PORT | 0 | 2 | | censys_ip, shodan_enterprise | -| ORG_STUB | 4 | 1 | dockerhub, github_org, google_playstore, postman | speculate | -| PASSWORD | 0 | 2 | | credshed, dehashed | -| PROTOCOL | 3 | 2 | legba, medusa, nmap_xml | censys_ip, fingerprintx | -| RAW_DNS_RECORD | 0 | 3 | | dnsbimi, dnsresolve, dnstlsrpt | -| RAW_TEXT | 2 | 1 | excavate, trufflehog | kreuzberg | -| SOCIAL | 7 | 4 | dockerhub, github_org, gitlab_com, gitlab_onprem, gowitness, postman, speculate | dockerhub, github_usersearch, gitlab_onprem, social | -| STORAGE_BUCKET | 9 | 6 | baddns_direct, bucket_amazon, bucket_digitalocean, bucket_file_enum, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, speculate | bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft | -| TECHNOLOGY | 5 | 10 | asset_inventory, gitlab_onprem, trajan, web_report, wpscan | ajaxpro, badsecrets, censys_ip, dotnetnuke, gitlab_onprem, gowitness, nuclei, shodan_enterprise, shodan_idb, wpscan | -| URL | 22 | 2 | ajaxpro, aspnet_bin_exposure, asset_inventory, baddns_direct, bypass403, generic_ssrf, git, gowitness, graphql_introspection, http, iis_shortnames, lightfuzz, ntlm, nuclei, portfilter, robots, speculate, telerik, url_manipulation, wafw00f, web_report, webbrute | gowitness, http | -| URL_HINT | 1 | 1 | webbrute_shortnames | iis_shortnames | -| URL_UNVERIFIED | 9 | 20 | code_repository, filedownload, http, oauth, portfilter, retirejs, social, speculate, trajan | azure_tenant, bevigil, bucket_file_enum, censys_ip, dnsbimi, dnscaa, dnstlsrpt, dockerhub, excavate, fingerprintx, github_codesearch, gowitness, hunterio, robots, securitytxt, urlscan, wayback, webbrute, webbrute_shortnames, wpscan | -| USERNAME | 1 | 2 | speculate | credshed, dehashed | -| WAF | 1 | 1 | asset_inventory | wafw00f | -| WEBSCREENSHOT | 0 | 1 | | gowitness | -| WEB_PARAMETER | 7 | 4 | hunt, lightfuzz, paramminer_cookies, paramminer_getparams, paramminer_headers, reflected_parameters, web_parameters | excavate, paramminer_cookies, paramminer_getparams, paramminer_headers | +| Event Type | # Consuming Modules | # Producing Modules | Consuming Modules | Producing Modules | +|---------------------|-----------------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| * | 24 | 0 | affiliates, cloudcheck, csv, discord, dnsresolve, elastic, json, kafka, mongo, mysql, nats, neo4j, postgres, python, rabbitmq, slack, splunk, sqlite, stdout, teams, txt, webhook, websocket, zeromq | | +| ASN | 0 | 1 | | asn | +| AZURE_TENANT | 1 | 1 | speculate | azure_tenant | +| CODE_REPOSITORY | 8 | 8 | docker_pull, git_clone, gitdumper, github_workflows, google_playstore, postman_download, trajan, trufflehog | code_repository, dockerhub, git, github_codesearch, github_org, gitlab_com, gitlab_onprem, postman | +| DNS_NAME | 57 | 39 | anubisdb, asset_inventory, azure_tenant, baddns, baddns_zone, bevigil, bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bufferoverrun, builtwith, c99, censys_dns, certspotter, chaos, credshed, crt, crt_db, dehashed, dnsbimi, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, emailformat, fullhunt, github_codesearch, github_usersearch, hackertarget, hunterio, leakix, myssl, nmap_xml, oauth, otx, pgp, portscan, rapiddns, securitytrails, securitytxt, shodan_dns, shodan_idb, skymem, speculate, subdomaincenter, subdomainradar, subdomains, trickest, urlscan, viewdns, virustotal, wayback | anubisdb, azure_tenant, bevigil, bufferoverrun, builtwith, c99, censys_dns, censys_ip, certspotter, chaos, crt, crt_db, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnsresolve, fullhunt, hackertarget, hunterio, leakix, myssl, ntlm, oauth, otx, rapiddns, securitytrails, shodan_dns, shodan_idb, speculate, sslcert, subdomaincenter, subdomainradar, trickest, urlscan, viewdns, virustotal, wayback | +| DNS_NAME_UNRESOLVED | 3 | 0 | baddns, speculate, subdomains | | +| DNS_NAME_UNVERIFIED | 0 | 1 | | virtualhost | +| EMAIL_ADDRESS | 1 | 11 | emails | credshed, dehashed, dnscaa, dnstlsrpt, emailformat, github_usersearch, hunterio, pgp, securitytxt, skymem, sslcert | +| FILESYSTEM | 4 | 9 | jadx, kreuzberg, trufflehog, unarchive | apkpure, docker_pull, filedownload, git_clone, gitdumper, github_workflows, jadx, postman_download, unarchive | +| FINDING | 2 | 38 | asset_inventory, web_report | ajaxpro, aspnet_bin_exposure, azure_tenant, baddns, baddns_direct, baddns_zone, badsecrets, bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bypass403, dotnetnuke, generic_ssrf, git, gitlab_onprem, graphql_introspection, host_header, hunt, legba, lightfuzz, medusa, newsletters, ntlm, nuclei, reflected_parameters, retirejs, shodan_enterprise, shodan_idb, speculate, telerik, trajan, trufflehog, url_manipulation, waf_bypass, wayback | +| GEOLOCATION | 0 | 2 | | ip2location, ipstack | +| HASHED_PASSWORD | 0 | 2 | | credshed, dehashed | +| HTTP_RESPONSE | 18 | 4 | ajaxpro, asset_inventory, badsecrets, dotnetnuke, excavate, filedownload, gitlab_onprem, host_header, newsletters, nmap_xml, ntlm, paramminer_cookies, paramminer_getparams, paramminer_headers, speculate, sslcert, telerik, trufflehog | http, lightfuzz, virtualhost, wayback | +| IP_ADDRESS | 11 | 5 | asn, asset_inventory, censys_ip, ip2location, ipneighbor, ipstack, nmap_xml, portscan, shodan_enterprise, shodan_idb, speculate | asset_inventory, censys_ip, dnsresolve, ipneighbor, speculate | +| IP_RANGE | 2 | 1 | portscan, speculate | dnsresolve | +| MOBILE_APP | 1 | 1 | apkpure | google_playstore | +| OPEN_TCP_PORT | 5 | 6 | asset_inventory, fingerprintx, http, nmap_xml, portfilter | asset_inventory, censys_ip, portscan, shodan_enterprise, shodan_idb, speculate | +| OPEN_UDP_PORT | 0 | 2 | | censys_ip, shodan_enterprise | +| ORG_STUB | 4 | 1 | dockerhub, github_org, google_playstore, postman | speculate | +| PASSWORD | 0 | 2 | | credshed, dehashed | +| PROTOCOL | 3 | 2 | legba, medusa, nmap_xml | censys_ip, fingerprintx | +| RAW_DNS_RECORD | 0 | 3 | | dnsbimi, dnsresolve, dnstlsrpt | +| RAW_TEXT | 2 | 1 | excavate, trufflehog | kreuzberg | +| SOCIAL | 7 | 4 | dockerhub, github_org, gitlab_com, gitlab_onprem, gowitness, postman, speculate | dockerhub, github_usersearch, gitlab_onprem, social | +| STORAGE_BUCKET | 9 | 6 | baddns_direct, bucket_amazon, bucket_digitalocean, bucket_file_enum, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, speculate | bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft | +| TECHNOLOGY | 4 | 9 | asset_inventory, gitlab_onprem, trajan, web_report | ajaxpro, badsecrets, censys_ip, dotnetnuke, gitlab_onprem, gowitness, nuclei, shodan_enterprise, shodan_idb | +| URL | 25 | 2 | ajaxpro, aspnet_bin_exposure, asset_inventory, baddns_direct, bypass403, generic_ssrf, git, gowitness, graphql_introspection, http, iis_shortnames, lightfuzz, ntlm, nuclei, portfilter, robots, speculate, telerik, url_manipulation, virtualhost, waf_bypass, wafw00f, wayback, web_report, webbrute | gowitness, http | +| URL_HINT | 1 | 1 | webbrute_shortnames | iis_shortnames | +| URL_UNVERIFIED | 9 | 19 | code_repository, filedownload, http, oauth, portfilter, retirejs, social, speculate, trajan | azure_tenant, bevigil, bucket_file_enum, censys_ip, dnsbimi, dnscaa, dnstlsrpt, dockerhub, excavate, fingerprintx, github_codesearch, gowitness, hunterio, robots, securitytxt, urlscan, wayback, webbrute, webbrute_shortnames | +| USERNAME | 1 | 2 | speculate | credshed, dehashed | +| VIRTUAL_HOST | 0 | 1 | | virtualhost | +| WAF | 1 | 1 | asset_inventory | wafw00f | +| WEBSCREENSHOT | 0 | 1 | | gowitness | +| WEB_PARAMETER | 7 | 5 | hunt, lightfuzz, paramminer_cookies, paramminer_getparams, paramminer_headers, reflected_parameters, web_parameters | excavate, paramminer_cookies, paramminer_getparams, paramminer_headers, wayback | <!-- END BBOT EVENTS --> [Next Up: Output -->](./output.md){ .md-button .md-button--primary } diff --git a/docs/scanning/index.md b/docs/scanning/index.md index 7cff759b8c..8fcc3870f4 100644 --- a/docs/scanning/index.md +++ b/docs/scanning/index.md @@ -187,35 +187,35 @@ A single module can have multiple flags. For example, the `securitytrails` modul ### List of Flags <!-- BBOT MODULE FLAGS --> -| Flag | # Modules | Description | Modules | -|------------------|-------------|----------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| safe | 99 | Non-intrusive and non-destructive | affiliates, aggregate, ajaxpro, anubisdb, apkpure, asn, aspnet_bin_exposure, azure_tenant, baddns, baddns_direct, baddns_zone, badsecrets, bevigil, bucket_amazon, bucket_digitalocean, bucket_file_enum, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bufferoverrun, builtwith, c99, censys_dns, censys_ip, certspotter, chaos, code_repository, credshed, crt, crt_db, dehashed, dnsbimi, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, docker_pull, dockerhub, emailformat, emails, excavate, filedownload, fingerprintx, fullhunt, git, git_clone, gitdumper, github_codesearch, github_org, github_usersearch, github_workflows, gitlab_com, gitlab_onprem, google_playstore, gowitness, graphql_introspection, hackertarget, http, hunt, hunterio, ip2location, ipstack, jadx, kreuzberg, leakix, myssl, newsletters, ntlm, oauth, otx, pgp, portfilter, postman, postman_download, rapiddns, reflected_parameters, retirejs, robots, securitytrails, securitytxt, shodan_dns, shodan_enterprise, shodan_idb, skymem, social, speculate, sslcert, subdomaincenter, subdomainradar, subdomains, trajan, trickest, trufflehog, unarchive, urlscan, viewdns, virustotal, wayback | -| passive | 68 | Never connects to target systems | affiliates, aggregate, anubisdb, apkpure, asn, azure_tenant, bevigil, bucket_file_enum, bufferoverrun, builtwith, c99, censys_dns, censys_ip, certspotter, chaos, code_repository, credshed, crt, crt_db, dehashed, dnsbimi, dnscaa, dnsdumpster, dnstlsrpt, docker_pull, dockerhub, emailformat, excavate, fullhunt, git_clone, gitdumper, github_codesearch, github_org, github_usersearch, github_workflows, google_playstore, hackertarget, hunterio, ip2location, ipneighbor, ipstack, jadx, kreuzberg, leakix, myssl, otx, pgp, portfilter, postman, postman_download, rapiddns, securitytrails, shodan_dns, shodan_enterprise, shodan_idb, skymem, social, speculate, subdomaincenter, subdomainradar, trajan, trickest, trufflehog, unarchive, urlscan, viewdns, virustotal, wayback | -| active | 51 | Makes active connections to target systems | ajaxpro, aspnet_bin_exposure, baddns, baddns_direct, baddns_zone, badsecrets, bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bypass403, dnsbrute, dnsbrute_mutations, dnscommonsrv, dotnetnuke, filedownload, fingerprintx, generic_ssrf, git, gitlab_com, gitlab_onprem, gowitness, graphql_introspection, host_header, http, hunt, iis_shortnames, legba, lightfuzz, medusa, newsletters, ntlm, nuclei, oauth, paramminer_cookies, paramminer_getparams, paramminer_headers, portscan, reflected_parameters, retirejs, robots, securitytxt, sslcert, telerik, url_manipulation, wafw00f, webbrute, webbrute_shortnames, wpscan | -| subdomain-enum | 47 | Enumerates subdomains | anubisdb, asn, azure_tenant, baddns_direct, baddns_zone, bevigil, bufferoverrun, builtwith, c99, censys_dns, certspotter, chaos, crt, crt_db, dnsbimi, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, fullhunt, github_codesearch, github_org, hackertarget, http, hunterio, ipneighbor, leakix, myssl, oauth, otx, postman, postman_download, rapiddns, securitytrails, securitytxt, shodan_dns, shodan_idb, sslcert, subdomaincenter, subdomainradar, subdomains, trickest, urlscan, virustotal, wayback | -| loud | 21 | Generates a large amount of network traffic | bypass403, dnsbrute, dnsbrute_mutations, dotnetnuke, host_header, iis_shortnames, ipneighbor, legba, lightfuzz, medusa, nuclei, paramminer_cookies, paramminer_getparams, paramminer_headers, portscan, telerik, url_manipulation, wafw00f, webbrute, webbrute_shortnames, wpscan | -| code-enum | 19 | Find public code repositories and search them for secrets etc. | apkpure, code_repository, docker_pull, dockerhub, git, git_clone, gitdumper, github_codesearch, github_org, github_usersearch, github_workflows, gitlab_com, gitlab_onprem, google_playstore, jadx, postman, postman_download, trajan, trufflehog | -| cloud-enum | 16 | Enumerates cloud resources | azure_tenant, baddns, baddns_direct, baddns_zone, bucket_amazon, bucket_digitalocean, bucket_file_enum, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, dnsbimi, dnstlsrpt, http, oauth, securitytxt | -| web | 16 | Non-intrusive web scan functionality | baddns, badsecrets, bucket_amazon, bucket_firebase, bucket_google, bucket_microsoft, filedownload, git, graphql_introspection, http, iis_shortnames, ntlm, oauth, robots, securitytxt, sslcert | -| web-heavy | 15 | More advanced web scanning functionality | ajaxpro, aspnet_bin_exposure, bucket_digitalocean, bucket_hetzner, bypass403, dotnetnuke, generic_ssrf, host_header, hunt, lightfuzz, reflected_parameters, retirejs, telerik, url_manipulation, webbrute_shortnames | -| slow | 10 | May take a long time to complete | bucket_digitalocean, bucket_hetzner, dnsbrute_mutations, docker_pull, fingerprintx, git_clone, gitdumper, paramminer_cookies, paramminer_getparams, paramminer_headers | -| email-enum | 9 | Enumerates email addresses | dehashed, dnscaa, dnstlsrpt, emailformat, emails, hunterio, pgp, skymem, sslcert | -| affiliates | 7 | Discovers affiliated hostnames/domains | affiliates, azure_tenant, builtwith, oauth, sslcert, trickest, viewdns | -| download | 7 | Modules that download files, apps, or repositories | apkpure, docker_pull, filedownload, git_clone, gitdumper, github_workflows, postman_download | -| invasive | 7 | Intrusive or potentially destructive | dotnetnuke, generic_ssrf, legba, lightfuzz, medusa, nuclei, telerik | -| baddns | 3 | Runs all modules from the DNS auditing tool BadDNS | baddns, baddns_direct, baddns_zone | -| web-paramminer | 3 | Discovers HTTP parameters through brute-force | paramminer_cookies, paramminer_getparams, paramminer_headers | -| iis-shortnames | 2 | Scans for IIS Shortname vulnerability | iis_shortnames, webbrute_shortnames | -| portscan | 2 | Discovers open ports | portscan, shodan_idb | -| social-enum | 2 | Enumerates social media | http, social | -| service-enum | 1 | Identifies protocols running on open ports | fingerprintx | -| subdomain-hijack | 1 | Detects hijackable subdomains | baddns | -| web-screenshots | 1 | Takes screenshots of web pages | gowitness | +| Flag | # Modules | Description | Modules | +|------------------|-------------|----------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| safe | 100 | Non-intrusive and non-destructive | affiliates, aggregate, ajaxpro, anubisdb, apkpure, asn, aspnet_bin_exposure, azure_tenant, baddns, baddns_direct, baddns_zone, badsecrets, bevigil, bucket_amazon, bucket_digitalocean, bucket_file_enum, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bufferoverrun, builtwith, c99, censys_dns, censys_ip, certspotter, chaos, code_repository, credshed, crt, crt_db, dehashed, dnsbimi, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, docker_pull, dockerhub, emailformat, emails, excavate, filedownload, fingerprintx, fullhunt, git, git_clone, gitdumper, github_codesearch, github_org, github_usersearch, github_workflows, gitlab_com, gitlab_onprem, google_playstore, gowitness, graphql_introspection, hackertarget, http, hunt, hunterio, ip2location, ipstack, jadx, kreuzberg, leakix, myssl, newsletters, ntlm, oauth, otx, pgp, portfilter, postman, postman_download, rapiddns, reflected_parameters, retirejs, robots, securitytrails, securitytxt, shodan_dns, shodan_enterprise, shodan_idb, skymem, social, speculate, sslcert, subdomaincenter, subdomainradar, subdomains, trajan, trickest, trufflehog, unarchive, urlscan, viewdns, virustotal, waf_bypass, wayback | +| passive | 68 | Never connects to target systems | affiliates, aggregate, anubisdb, apkpure, asn, azure_tenant, bevigil, bucket_file_enum, bufferoverrun, builtwith, c99, censys_dns, censys_ip, certspotter, chaos, code_repository, credshed, crt, crt_db, dehashed, dnsbimi, dnscaa, dnsdumpster, dnstlsrpt, docker_pull, dockerhub, emailformat, excavate, fullhunt, git_clone, gitdumper, github_codesearch, github_org, github_usersearch, github_workflows, google_playstore, hackertarget, hunterio, ip2location, ipneighbor, ipstack, jadx, kreuzberg, leakix, myssl, otx, pgp, portfilter, postman, postman_download, rapiddns, securitytrails, shodan_dns, shodan_enterprise, shodan_idb, skymem, social, speculate, subdomaincenter, subdomainradar, trajan, trickest, trufflehog, unarchive, urlscan, viewdns, virustotal, wayback | +| active | 52 | Makes active connections to target systems | ajaxpro, aspnet_bin_exposure, baddns, baddns_direct, baddns_zone, badsecrets, bucket_amazon, bucket_digitalocean, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, bypass403, dnsbrute, dnsbrute_mutations, dnscommonsrv, dotnetnuke, filedownload, fingerprintx, generic_ssrf, git, gitlab_com, gitlab_onprem, gowitness, graphql_introspection, host_header, http, hunt, iis_shortnames, legba, lightfuzz, medusa, newsletters, ntlm, nuclei, oauth, paramminer_cookies, paramminer_getparams, paramminer_headers, portscan, reflected_parameters, retirejs, robots, securitytxt, sslcert, telerik, url_manipulation, virtualhost, waf_bypass, wafw00f, webbrute, webbrute_shortnames | +| subdomain-enum | 47 | Enumerates subdomains | anubisdb, asn, azure_tenant, baddns_direct, baddns_zone, bevigil, bufferoverrun, builtwith, c99, censys_dns, certspotter, chaos, crt, crt_db, dnsbimi, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, fullhunt, github_codesearch, github_org, hackertarget, http, hunterio, ipneighbor, leakix, myssl, oauth, otx, postman, postman_download, rapiddns, securitytrails, securitytxt, shodan_dns, shodan_idb, sslcert, subdomaincenter, subdomainradar, subdomains, trickest, urlscan, virustotal, wayback | +| loud | 21 | Generates a large amount of network traffic | bypass403, dnsbrute, dnsbrute_mutations, dotnetnuke, host_header, iis_shortnames, ipneighbor, legba, lightfuzz, medusa, nuclei, paramminer_cookies, paramminer_getparams, paramminer_headers, portscan, telerik, url_manipulation, virtualhost, wafw00f, webbrute, webbrute_shortnames | +| code-enum | 19 | Find public code repositories and search them for secrets etc. | apkpure, code_repository, docker_pull, dockerhub, git, git_clone, gitdumper, github_codesearch, github_org, github_usersearch, github_workflows, gitlab_com, gitlab_onprem, google_playstore, jadx, postman, postman_download, trajan, trufflehog | +| cloud-enum | 16 | Enumerates cloud resources | azure_tenant, baddns, baddns_direct, baddns_zone, bucket_amazon, bucket_digitalocean, bucket_file_enum, bucket_firebase, bucket_google, bucket_hetzner, bucket_microsoft, dnsbimi, dnstlsrpt, http, oauth, securitytxt | +| web | 16 | Non-intrusive web scan functionality | baddns, badsecrets, bucket_amazon, bucket_firebase, bucket_google, bucket_microsoft, filedownload, git, graphql_introspection, http, iis_shortnames, ntlm, oauth, robots, securitytxt, sslcert | +| web-heavy | 16 | More advanced web scanning functionality | ajaxpro, aspnet_bin_exposure, bucket_digitalocean, bucket_hetzner, bypass403, dotnetnuke, generic_ssrf, host_header, hunt, lightfuzz, reflected_parameters, retirejs, telerik, url_manipulation, waf_bypass, webbrute_shortnames | +| slow | 11 | May take a long time to complete | bucket_digitalocean, bucket_hetzner, dnsbrute_mutations, docker_pull, fingerprintx, git_clone, gitdumper, paramminer_cookies, paramminer_getparams, paramminer_headers, virtualhost | +| email-enum | 9 | Enumerates email addresses | dehashed, dnscaa, dnstlsrpt, emailformat, emails, hunterio, pgp, skymem, sslcert | +| affiliates | 7 | Discovers affiliated hostnames/domains | affiliates, azure_tenant, builtwith, oauth, sslcert, trickest, viewdns | +| download | 7 | Modules that download files, apps, or repositories | apkpure, docker_pull, filedownload, git_clone, gitdumper, github_workflows, postman_download | +| invasive | 7 | Intrusive or potentially destructive | dotnetnuke, generic_ssrf, legba, lightfuzz, medusa, nuclei, telerik | +| baddns | 3 | Runs all modules from the DNS auditing tool BadDNS | baddns, baddns_direct, baddns_zone | +| web-paramminer | 3 | Discovers HTTP parameters through brute-force | paramminer_cookies, paramminer_getparams, paramminer_headers | +| iis-shortnames | 2 | Scans for IIS Shortname vulnerability | iis_shortnames, webbrute_shortnames | +| portscan | 2 | Discovers open ports | portscan, shodan_idb | +| social-enum | 2 | Enumerates social media | http, social | +| service-enum | 1 | Identifies protocols running on open ports | fingerprintx | +| subdomain-hijack | 1 | Detects hijackable subdomains | baddns | +| web-screenshots | 1 | Takes screenshots of web pages | gowitness | <!-- END BBOT MODULE FLAGS --> ## Dependencies -BBOT modules have external dependencies ranging from OS packages (`openssl`) to binaries (`nuclei`) to Python libraries (`wappalyzer`). When a module is enabled, installation of its dependencies happens at runtime with [Ansible](https://github.com/ansible/ansible). BBOT provides several command-line flags to control how dependencies are installed. +BBOT modules have external dependencies ranging from OS packages (`openssl`) to binaries (`nuclei`) to Python libraries (`pyOpenSSL`). When a module is enabled, installation of its dependencies happens at runtime with [Ansible](https://github.com/ansible/ansible). BBOT provides several command-line flags to control how dependencies are installed. - `--no-deps` - Don't install module dependencies - `--force-deps` - Force install all module dependencies diff --git a/docs/scanning/output.md b/docs/scanning/output.md index 91c8a9561b..8715698ed3 100644 --- a/docs/scanning/output.md +++ b/docs/scanning/output.md @@ -46,13 +46,13 @@ exclude_output_modules: The `stdout` output module is what you see when you execute BBOT in the terminal. By default it looks the same as the [`txt`](#txt) module, but it has options you can customize. You can filter by event type, choose the data format (`text`, `json`), and which fields you want to see: <!-- BBOT MODULE OPTIONS STDOUT --> -| Config Option | Type | Description | Default | -|------------------------------|--------|--------------------------------------------------|-----------| -| modules.stdout.accept_dupes | bool | Whether to show duplicate events, default True | True | -| modules.stdout.event_fields | list | Which event fields to display | [] | -| modules.stdout.event_types | list | Which events to display, default all event types | [] | -| modules.stdout.format | str | Which text format to display, choices: text,json | text | -| modules.stdout.in_scope_only | bool | Whether to only show in-scope events | False | +| Config Option | Type | Description | Default | +|------------------------------|-------------------------|--------------------------------------------------|-----------| +| modules.stdout.accept_dupes | bool | Whether to show duplicate events, default True | True | +| modules.stdout.event_fields | list | Which event fields to display | [] | +| modules.stdout.event_types | list | Which events to display, default all event types | [] | +| modules.stdout.format | Literal['text', 'json'] | Which text format to display, choices: text,json | text | +| modules.stdout.in_scope_only | bool | Whether to only show in-scope events | False | <!-- END BBOT MODULE OPTIONS STDOUT --> ### TXT diff --git a/docs/scanning/presets_list.md b/docs/scanning/presets_list.md index 1edd6fa6e2..419a99a808 100644 --- a/docs/scanning/presets_list.md +++ b/docs/scanning/presets_list.md @@ -116,6 +116,7 @@ Recursive web directory brute-force (aggressive) # we exploit the shortnames vulnerability to produce URL_HINTs which are consumed by webbrute_shortnames detect_only: False webbrute: + avoid_wafs: False max_depth: 3 lines: 5000 extensions: @@ -282,7 +283,7 @@ Everything everywhere all at once ??? note "`kitchen-sink.yml`" ```yaml title="~/.bbot/presets/kitchen-sink.yml" description: Everything everywhere all at once - + include: - subdomain-enum - cloud-enum @@ -294,7 +295,7 @@ Everything everywhere all at once - dirbust-light - web-screenshots - baddns-heavy - + config: modules: baddns: @@ -303,6 +304,8 @@ Everything everywhere all at once recursive_mutations: true dnscommonsrv: recursive_mutations: true + webbrute: + avoid_wafs: False wayback: urls: True parameters: True @@ -353,11 +356,11 @@ Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force paramet flags: - web-paramminer - + modules: - robots - wayback - + config: modules: lightfuzz: @@ -788,25 +791,121 @@ Detect technologies via Nuclei, and FingerprintX Modules: [0]("") -## **test** +## **virtualhost** -Detect technologies via Nuclei, and FingerprintX +Virtual host discovery: subdomain brute-force and mutations against the target host's Host header / SNI. -??? note "`test.yml`" - ```yaml title="~/.bbot/presets/test.yml" - description: Detect technologies via Nuclei, and FingerprintX +??? note "`virtualhost.yml`" + ```yaml title="~/.bbot/presets/web/virtualhost.yml" + description: "Virtual host discovery: subdomain brute-force and mutations against the target host's Host header / SNI." modules: - - nuclei - - fingerprintx + - virtualhost + ``` + +Category: web + +Modules: [0]("") + +## **virtualhost-heavy** + +Aggressive virtual host discovery: everything in virtualhost, plus special-host probing, certificate SAN extraction, and wordcloud-driven candidate testing. + +??? note "`virtualhost-heavy.yml`" + ```yaml title="~/.bbot/presets/web/virtualhost-heavy.yml" + description: "Aggressive virtual host discovery: everything in virtualhost, plus special-host probing, certificate SAN extraction, and wordcloud-driven candidate testing." - target: - - tesasdft.txt + include: + - virtualhost config: modules: - nuclei: - tags: tech + virtualhost: + special_hosts: True + certificate_sans: True + wordcloud_check: True + ``` + +Category: web + +Modules: [0]("") + +## **waf-bypass** + +WAF bypass detection with subdomain enumeration + +??? note "`waf-bypass.yml`" + ```yaml title="~/.bbot/presets/waf-bypass.yml" + description: WAF bypass detection with subdomain enumeration + + flags: + # enable subdomain enumeration to find potential bypass targets + - subdomain-enum + + modules: + # explicitly enable the waf_bypass module for detection + - waf_bypass + # ensure http is enabled for web probing + - http + + config: + # waf_bypass module configuration + modules: + waf_bypass: + similarity_threshold: 0.90 + search_ip_neighbors: true + neighbor_cidr: 24 + ``` + + + +Modules: [0]("") + +## **wayback** + +Discover URLs and interesting archived files via the Wayback Machine + +??? note "`wayback.yml`" + ```yaml title="~/.bbot/presets/wayback.yml" + description: Discover URLs and interesting archived files via the Wayback Machine + + include: + - subdomain-enum + + modules: + - wayback + + config: + modules: + wayback: + urls: True + ``` + + + +Modules: [0]("") + +## **wayback-heavy** + +Full Wayback Machine integration - URL discovery, parameter extraction, archived page retrieval, and interesting file detection + +??? note "`wayback-heavy.yml`" + ```yaml title="~/.bbot/presets/wayback-heavy.yml" + description: Full Wayback Machine integration - URL discovery, parameter extraction, archived page retrieval, and interesting file detection + + include: + - subdomain-enum + + modules: + - wayback + - badsecrets + + config: + modules: + wayback: + urls: True + parameters: True + archive: True ``` @@ -876,78 +975,7 @@ Take screenshots of webpages -Modules: [3]("`gowitness`, `httpx`, `social`") - -## **web-thorough** - -Aggressive web scan - -??? note "`web-thorough.yml`" - ```yaml title="~/.bbot/presets/web-thorough.yml" - description: Aggressive web scan - - include: - # include the web-basic preset - - web-basic - - flags: - - web-thorough - ``` - - - -Modules: [32]("`ajaxpro`, `aspnet_bin_exposure`, `azure_realm`, `baddns`, `badsecrets`, `bucket_amazon`, `bucket_digitalocean`, `bucket_firebase`, `bucket_google`, `bucket_microsoft`, `bypass403`, `dotnetnuke`, `ffuf_shortnames`, `filedownload`, `generic_ssrf`, `git`, `graphql_introspection`, `host_header`, `httpx`, `hunt`, `iis_shortnames`, `lightfuzz`, `ntlm`, `oauth`, `reflected_parameters`, `retirejs`, `robots`, `securitytxt`, `smuggler`, `sslcert`, `telerik`, `url_manipulation`") - -## **wayback** - -Discover URLs and interesting archived files via the Wayback Machine - -??? note "`wayback.yml`" - ```yaml title="~/.bbot/presets/wayback.yml" - description: Discover URLs and interesting archived files via the Wayback Machine - - include: - - subdomain-enum - - modules: - - wayback - - config: - modules: - wayback: - urls: True - ``` - - - -Modules: [52]("`anubisdb`, `asn`, `azure_realm`, `azure_tenant`, `baddns_direct`, `baddns_zone`, `bevigil`, `bufferoverrun`, `builtwith`, `c99`, `censys_dns`, `certspotter`, `chaos`, `crt`, `crt_db`, `digitorus`, `dnsbimi`, `dnsbrute`, `dnsbrute_mutations`, `dnscaa`, `dnscommonsrv`, `dnsdumpster`, `dnstlsrpt`, `fullhunt`, `github_codesearch`, `github_org`, `hackertarget`, `httpx`, `hunterio`, `ipneighbor`, `leakix`, `myssl`, `oauth`, `otx`, `passivetotal`, `postman`, `postman_download`, `rapiddns`, `securitytrails`, `securitytxt`, `shodan_dns`, `shodan_idb`, `sitedossier`, `social`, `sslcert`, `subdomaincenter`, `subdomainradar`, `trickest`, `urlscan`, `virustotal`, `wayback`, `httpx`") - -## **wayback-heavy** - -Full Wayback Machine integration - URL discovery, parameter extraction, archived page retrieval, and interesting file detection - -??? note "`wayback-heavy.yml`" - ```yaml title="~/.bbot/presets/wayback-heavy.yml" - description: Full Wayback Machine integration - URL discovery, parameter extraction, archived page retrieval, and interesting file detection - - include: - - subdomain-enum - - modules: - - wayback - - badsecrets - - config: - modules: - wayback: - urls: True - parameters: True - archive: True - ``` - - - -Modules: [53]("`anubisdb`, `asn`, `azure_realm`, `azure_tenant`, `baddns_direct`, `baddns_zone`, `badsecrets`, `bevigil`, `bufferoverrun`, `builtwith`, `c99`, `censys_dns`, `certspotter`, `chaos`, `crt`, `crt_db`, `digitorus`, `dnsbimi`, `dnsbrute`, `dnsbrute_mutations`, `dnscaa`, `dnscommonsrv`, `dnsdumpster`, `dnstlsrpt`, `fullhunt`, `github_codesearch`, `github_org`, `hackertarget`, `httpx`, `hunterio`, `ipneighbor`, `leakix`, `myssl`, `oauth`, `otx`, `passivetotal`, `postman`, `postman_download`, `rapiddns`, `securitytrails`, `securitytxt`, `shodan_dns`, `shodan_idb`, `sitedossier`, `social`, `sslcert`, `subdomaincenter`, `subdomainradar`, `trickest`, `urlscan`, `virustotal`, `wayback`, `httpx`") +Modules: [0]("") <!-- END BBOT PRESET YAML --> ## Table of Default Presets @@ -969,9 +997,9 @@ Here is a the same data, but in a table: | iis-shortnames | web | Recursively enumerate IIS shortnames | 0 | | | kitchen-sink | | Everything everywhere all at once | 7 | baddns, baddns_direct, baddns_zone, http, hunt, reflected_parameters, webbrute | | lightfuzz | web | Default fuzzing: all 9 submodules (cmdi, crypto, path, serial, sqli, ssti, xss, esi, ssrf) plus companion modules (badsecrets, hunt, reflected_parameters). POST fuzzing disabled but try_post_as_get enabled, so POST params are retested as GET. Skips confirmed WAFs. | 6 | badsecrets, http, hunt, lightfuzz, portfilter, reflected_parameters | -| lightfuzz-heavy | web | Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force parameter discovery (headers, GET params, cookies), POST request fuzzing enabled, try_get_as_post enabled (GET params retested as POST), and robots.txt parsing. Still skips confirmed WAFs. | 7 | badsecrets, http, hunt, lightfuzz, portfilter, reflected_parameters, robots | +| lightfuzz-heavy | web | Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force parameter discovery (headers, GET params, cookies), POST request fuzzing enabled, try_get_as_post enabled (GET params retested as POST), and robots.txt parsing. Still skips confirmed WAFs. | 8 | badsecrets, http, hunt, lightfuzz, portfilter, reflected_parameters, robots, wayback | | lightfuzz-light | web | Minimal fuzzing: only path traversal, SQLi, and XSS submodules. No POST requests. No companion modules. Safest option for running alongside larger scans with minimal overhead. | 3 | http, lightfuzz, portfilter | -| lightfuzz-max | web | Maximum fuzzing: everything in lightfuzz-heavy, plus the heavy paramminer variant (1-3 letter brute-force on GET params, case mutation on case-sensitive backends, recycle_words on all paramminer modules), WAF targets are no longer skipped, each unique parameter-value pair is fuzzed individually (no collapsing), common headers like X-Forwarded-For are fuzzed even if not observed, and potential parameters are speculated from JSON/XML response bodies. Significantly increases scan time. | 7 | badsecrets, http, hunt, lightfuzz, portfilter, reflected_parameters, robots | +| lightfuzz-max | web | Maximum fuzzing: everything in lightfuzz-heavy, plus the heavy paramminer variant (1-3 letter brute-force on GET params, case mutation on case-sensitive backends, recycle_words on all paramminer modules), WAF targets are no longer skipped, each unique parameter-value pair is fuzzed individually (no collapsing), common headers like X-Forwarded-For are fuzzed even if not observed, and potential parameters are speculated from JSON/XML response bodies. Significantly increases scan time. | 8 | badsecrets, http, hunt, lightfuzz, portfilter, reflected_parameters, robots, wayback | | lightfuzz-xss | web | XSS-only: enables only the xss submodule with paramminer_getparams and reflected_parameters. POST disabled, no query string collapsing. Example of a focused single-submodule preset. | 5 | http, lightfuzz, paramminer_getparams, portfilter, reflected_parameters | | nuclei | nuclei | Run nuclei scans against all discovered targets | 3 | http, nuclei, portfilter | | nuclei-budget | nuclei | Run nuclei scans against all discovered targets, using budget mode to look for low hanging fruit with greatly reduced number of requests | 3 | http, nuclei, portfilter | @@ -983,9 +1011,11 @@ Here is a the same data, but in a table: | spider-heavy | | Recursive web spider with more aggressive settings | 1 | http | | subdomain-enum | | Enumerate subdomains via APIs, brute-force | 0 | | | tech-detect | | Detect technologies via Nuclei, and FingerprintX | 2 | fingerprintx, nuclei | -| test | | Detect technologies via Nuclei, and FingerprintX | 2 | fingerprintx, nuclei | -| wayback | | Discover URLs and interesting archived files via the Wayback Machine | 52 | anubisdb, asn, azure_realm, azure_tenant, baddns_direct, baddns_zone, bevigil, bufferoverrun, builtwith, c99, censys_dns, certspotter, chaos, crt, crt_db, digitorus, dnsbimi, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, fullhunt, github_codesearch, github_org, hackertarget, httpx, hunterio, ipneighbor, leakix, myssl, oauth, otx, passivetotal, postman, postman_download, rapiddns, securitytrails, securitytxt, shodan_dns, shodan_idb, sitedossier, social, sslcert, subdomaincenter, subdomainradar, trickest, urlscan, virustotal, wayback | -| wayback-heavy | | Full Wayback Machine integration - URL discovery, parameter extraction, archived page retrieval, and interesting file detection | 53 | anubisdb, asn, azure_realm, azure_tenant, baddns_direct, baddns_zone, badsecrets, bevigil, bufferoverrun, builtwith, c99, censys_dns, certspotter, chaos, crt, crt_db, digitorus, dnsbimi, dnsbrute, dnsbrute_mutations, dnscaa, dnscommonsrv, dnsdumpster, dnstlsrpt, fullhunt, github_codesearch, github_org, hackertarget, httpx, hunterio, ipneighbor, leakix, myssl, oauth, otx, passivetotal, postman, postman_download, rapiddns, securitytrails, securitytxt, shodan_dns, shodan_idb, sitedossier, social, sslcert, subdomaincenter, subdomainradar, trickest, urlscan, virustotal, wayback | +| virtualhost | web | Virtual host discovery: subdomain brute-force and mutations against the target host's Host header / SNI. | 1 | virtualhost | +| virtualhost-heavy | web | Aggressive virtual host discovery: everything in virtualhost, plus special-host probing, certificate SAN extraction, and wordcloud-driven candidate testing. | 1 | virtualhost | +| waf-bypass | | WAF bypass detection with subdomain enumeration | 2 | http, waf_bypass | +| wayback | | Discover URLs and interesting archived files via the Wayback Machine | 1 | wayback | +| wayback-heavy | | Full Wayback Machine integration - URL discovery, parameter extraction, archived page retrieval, and interesting file detection | 2 | badsecrets, wayback | | web | | Quick web scan | 0 | | | web-heavy | | Aggressive web scan | 0 | | | web-screenshots | | Take screenshots of webpages | 0 | | From b909889cb10d022fbdf629458ffe72bd10af09e4 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 18:25:16 -0400 Subject: [PATCH 24/29] Fix stale 2.x references in prose docs Drop removed VULNERABILITY event type, --allow-deadly flag, and httpx module reference; rename nuclei-intense/lightfuzz-superheavy presets to nuclei-heavy/lightfuzz-max; INFORMATIONAL severity to INFO; wappalyzer dep example to pyOpenSSL. --- AGENTS.md | 2 +- docs/modules/lightfuzz.md | 8 ++++---- docs/modules/wayback.md | 12 ++++++------ docs/scanning/tips_and_tricks.md | 2 +- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6c34ed62d7..c86e5cf4e1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -121,7 +121,7 @@ BBOT is an async, recursive OSINT tool. A scan starts with **seed events** (targ Events are the currency of BBOT. Every piece of data -- a hostname, IP, URL, open port, finding -- is an event. Events have: -- **type**: `DNS_NAME`, `IP_ADDRESS`, `URL`, `OPEN_TCP_PORT`, `HTTP_RESPONSE`, `FINDING`, `VULNERABILITY`, `EMAIL_ADDRESS`, etc. +- **type**: `DNS_NAME`, `IP_ADDRESS`, `URL`, `OPEN_TCP_PORT`, `HTTP_RESPONSE`, `FINDING`, `EMAIL_ADDRESS`, etc. - **data**: the actual data (a string, dict, etc.) - **parent**: the event that led to this one (forming a discovery chain) - **scope_distance**: how many hops from the original target (0 = in-scope) diff --git a/docs/modules/lightfuzz.md b/docs/modules/lightfuzz.md index 14d078fa32..6aa7c1bff7 100644 --- a/docs/modules/lightfuzz.md +++ b/docs/modules/lightfuzz.md @@ -36,7 +36,7 @@ Severity represents how bad the issue is *if it's real*. Levels, from worst to l | **HIGH** | Significant security impact (e.g., SQL injection, SSRF, path traversal, unsafe deserialization) | | **MEDIUM** | Moderate impact, often client-side (e.g., XSS, ESI) | | **LOW** | Minor or limited-scope issue | -| **INFORMATIONAL** | Interesting observation, not directly exploitable (e.g., cryptographic parameter detected) | +| **INFO** | Interesting observation, not directly exploitable (e.g., cryptographic parameter detected) | ### Confidence @@ -159,13 +159,13 @@ This stage bypasses the entropy gate (Stage 1) because the short ASCII plaintext **Stage 1 — Cryptanalysis Gate**: Checks if the parameter value is likely encrypted by calculating Shannon entropy (threshold: 4.5) and whether its decoded length is a multiple of 8 (suggesting a block cipher). If entropy is below threshold, all per-value tests are skipped (Stage 0's cross-value check has already run). -**Stage 2 — Response Divergence** (INFORMATIONAL severity, LOW confidence): Performs byte-level manipulations (truncation and single-byte mutation) and compares responses against both the baseline and an arbitrary garbage value. If the manipulated ciphertext produces a *different* response from both the original *and* garbage input, the parameter likely drives a real cryptographic operation. +**Stage 2 — Response Divergence** (INFO severity, LOW confidence): Performs byte-level manipulations (truncation and single-byte mutation) and compares responses against both the baseline and an arbitrary garbage value. If the manipulated ciphertext produces a *different* response from both the original *and* garbage input, the parameter likely drives a real cryptographic operation. -**Stage 3 — Error String Detection** (INFORMATIONAL severity, LOW confidence): Scans manipulation responses for cryptographic error messages using YARA rules (e.g., "padding is invalid", "invalid mac", "OpenSSL Error"). Errors present in the baseline are filtered out to avoid false positives. +**Stage 3 — Error String Detection** (INFO severity, LOW confidence): Scans manipulation responses for cryptographic error messages using YARA rules (e.g., "padding is invalid", "invalid mac", "OpenSSL Error"). Errors present in the baseline are filtered out to avoid false positives. **Stage 4 — Padding Oracle** (HIGH severity, HIGH confidence): If a block cipher is suspected, performs a targeted padding oracle test. Constructs a crafted ciphertext with a null IV block and iterates through all 256 possible last-byte values. A true padding oracle produces a small number of differing responses (1 up to block_size, since multi-byte padding values like `\x02\x02` can also produce valid padding if the intermediate bytes happen to align). To avoid false positives from servers that reflect or reveal submitted/decrypted values, probe values are stripped from both responses before comparison, and small character-level differences (≤5 chars in equal-length responses) are tolerated. Handles the edge case where the baseline byte is the correct padding byte (1/255 chance) by retrying with a different baseline. -**Stage 5 — Hash Length Extension** (INFORMATIONAL severity, LOW confidence): If the parameter value matches a known hash length (MD5/SHA-1/SHA-256/SHA-384/SHA-512), checks whether modifying *other* parameters on the same request causes the hash parameter's response to change — suggesting those parameters are inputs to the hash, which could enable length extension attacks. +**Stage 5 — Hash Length Extension** (INFO severity, LOW confidence): If the parameter value matches a known hash length (MD5/SHA-1/SHA-256/SHA-384/SHA-512), checks whether modifying *other* parameters on the same request causes the hash parameter's response to change — suggesting those parameters are inputs to the hash, which could enable length extension attacks. ### `serial` — Unsafe Deserialization diff --git a/docs/modules/wayback.md b/docs/modules/wayback.md index a3f30f3e23..972949cb53 100644 --- a/docs/modules/wayback.md +++ b/docs/modules/wayback.md @@ -27,7 +27,7 @@ To unlock the more advanced features, you need to enable them via configuration ### URL Discovery (`urls: True`) -When `urls` is enabled, wayback emits `URL_UNVERIFIED` events for every unique URL found in the Wayback Machine's index. These are tagged with `from-wayback` and sent through BBOT's normal URL verification pipeline (httpx). +When `urls` is enabled, wayback emits `URL_UNVERIFIED` events for every unique URL found in the Wayback Machine's index. These are tagged with `from-wayback` and sent through BBOT's normal URL verification pipeline (the `http` module). Before emission, URLs go through several cleanup steps: @@ -39,7 +39,7 @@ Before emission, URLs go through several cleanup steps: When `parameters` is enabled (requires `urls: True`), wayback extracts query string parameters from archived URLs and emits them as `WEB_PARAMETER` events. This is useful for discovering GET parameters that can be fed into fuzzing modules like lightfuzz. -Parameters are cached and only emitted after the corresponding URL has been verified as live by httpx. This prevents emitting parameters for URLs that no longer exist. +Parameters are cached and only emitted after the corresponding URL has been verified as live by the `http` module. This prevents emitting parameters for URLs that no longer exist. !!! note Parameter extraction requires at least one module that consumes `WEB_PARAMETER` events to be active (e.g. `lightfuzz`, `hunt`, `paramminer_getparams`). If no such module is present, parameter extraction is automatically disabled with a warning. @@ -99,9 +99,9 @@ Wayback's extended features are also enabled in several other presets: |-----------------------|-----------------------------------------| | `kitchen-sink` | `urls`, `parameters`, `archive` | | `dirbust-heavy` | `urls` | -| `nuclei-intense` | `urls` | +| `nuclei-heavy` | `urls` | | `lightfuzz-heavy` | `urls`, `parameters` | -| `lightfuzz-superheavy`| `urls`, `parameters`, `archive` | +| `lightfuzz-max` | `urls`, `parameters`, `archive` | ## Example Commands @@ -122,12 +122,12 @@ bbot -p wayback-heavy -t evilcorp.com ```bash # Enable wayback URLs alongside a nuclei scan -bbot -p nuclei -m wayback -c modules.wayback.urls=True --allow-deadly -t evilcorp.com +bbot -p nuclei -m wayback -c modules.wayback.urls=True -t evilcorp.com ``` ```bash # Pair with lightfuzz for parameter fuzzing using archived parameters -bbot -p lightfuzz-heavy spider -t evilcorp.com --allow-deadly +bbot -p lightfuzz-heavy spider -t evilcorp.com ``` ```bash diff --git a/docs/scanning/tips_and_tricks.md b/docs/scanning/tips_and_tricks.md index 81f34f0ad4..ad6d6a05b3 100644 --- a/docs/scanning/tips_and_tricks.md +++ b/docs/scanning/tips_and_tricks.md @@ -161,7 +161,7 @@ If you already have a list of discovered targets (e.g. URLs) and don't need DNS- bbot -m http gowitness -t urls.txt -c dns.disable=true ~~~ -Note that the above setting _completely_ disables DNS, meaning even `A` and `AAAA` records are not resolved. This can cause problems if you're using an IP whitelist or blacklist. In this case, you'll want to use `dns.minimal` instead: +Note that the above setting _completely_ disables DNS, meaning even `A` and `AAAA` records are not resolved. This can cause problems if you're using an IP-based target or blacklist. In this case, you'll want to use `dns.minimal` instead: ~~~bash # only resolve A and AAAA records From 353318a9c4b65450a1be58d1fa69fb6a0056916f Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 19:18:27 -0400 Subject: [PATCH 25/29] Keep metavar placeholders in CLI help Regenerate docs/scanning/advanced.md under Python 3.12. Python 3.13+ argparse collapses repeated metavars (-H, --custom-headers CUSTOM_HEADERS), dropping the placeholder after the short flag. --- docs/scanning/advanced.md | 48 ++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/docs/scanning/advanced.md b/docs/scanning/advanced.md index 7f8dd9c089..c9e56f7be4 100644 --- a/docs/scanning/advanced.md +++ b/docs/scanning/advanced.md @@ -43,9 +43,9 @@ usage: bbot [-h] [-t TARGET [TARGET ...]] [-s SEEDS [SEEDS ...]] [-om MODULE [MODULE ...]] [-eom MODULE [MODULE ...]] [-lo] [--json] [--brief] [--no-color] [--event-types EVENT_TYPES [EVENT_TYPES ...]] [--exclude-cdn] - [--no-deps | --force-deps | --retry-deps | - --ignore-failed-deps] [--install-all-deps] [--version] - [--reset-config] [--reset-secrets] [--proxy HTTP_PROXY] + [--no-deps | --force-deps | --retry-deps | --ignore-failed-deps] + [--install-all-deps] [--version] [--reset-config] + [--reset-secrets] [--proxy HTTP_PROXY] [--no-proxy HOST [HOST ...]] [-H CUSTOM_HEADERS [CUSTOM_HEADERS ...]] [-C CUSTOM_COOKIES [CUSTOM_COOKIES ...]] @@ -58,39 +58,40 @@ options: -h, --help show this help message and exit Target: - -t, --targets TARGET [TARGET ...] + -t TARGET [TARGET ...], --targets TARGET [TARGET ...] Target scope - -s, --seeds SEEDS [SEEDS ...] + -s SEEDS [SEEDS ...], --seeds SEEDS [SEEDS ...] Define seeds to drive passive modules without being in scope (if not specified, defaults to same as targets) - -b, --blacklist BLACKLIST [BLACKLIST ...] + -b BLACKLIST [BLACKLIST ...], --blacklist BLACKLIST [BLACKLIST ...] Don't touch these things --strict-scope Don't consider subdomains of target to be in-scope - exact matches only Presets: - -p, --preset [PRESET ...] + -p [PRESET ...], --preset [PRESET ...] Enable BBOT preset(s) - -c, --config [CONFIG ...] + -c [CONFIG ...], --config [CONFIG ...] Custom config options in key=value format: e.g. 'modules.shodan.api_key=1234' -lp, --list-presets List available presets. Modules: - -m, --modules MODULE [MODULE ...] + -m MODULE [MODULE ...], --modules MODULE [MODULE ...] Modules to enable. Choices: affiliates,ajaxpro,anubisdb,apkpure,asn,aspnet_bin_exposure,azure_tenant,baddns,baddns_direct,baddns_zone,badsecrets,bevigil,bucket_amazon,bucket_digitalocean,bucket_file_enum,bucket_firebase,bucket_google,bucket_hetzner,bucket_microsoft,bufferoverrun,builtwith,bypass403,c99,censys_dns,censys_ip,certspotter,chaos,code_repository,credshed,crt,crt_db,dehashed,dnsbimi,dnsbrute,dnsbrute_mutations,dnscaa,dnscommonsrv,dnsdumpster,dnstlsrpt,docker_pull,dockerhub,dotnetnuke,emailformat,filedownload,fingerprintx,fullhunt,generic_ssrf,git,git_clone,gitdumper,github_codesearch,github_org,github_usersearch,github_workflows,gitlab_com,gitlab_onprem,google_playstore,gowitness,graphql_introspection,hackertarget,host_header,http,hunt,hunterio,iis_shortnames,ip2location,ipneighbor,ipstack,jadx,kreuzberg,leakix,legba,lightfuzz,medusa,myssl,newsletters,ntlm,nuclei,oauth,otx,paramminer_cookies,paramminer_getparams,paramminer_headers,pgp,portfilter,portscan,postman,postman_download,rapiddns,reflected_parameters,retirejs,robots,securitytrails,securitytxt,shodan_dns,shodan_enterprise,shodan_idb,skymem,social,sslcert,subdomaincenter,subdomainradar,telerik,trajan,trickest,trufflehog,url_manipulation,urlscan,viewdns,virtualhost,virustotal,waf_bypass,wafw00f,wayback,webbrute,webbrute_shortnames -l, --list-modules List available modules. -lmo, --list-module-options Show all module config options - -em, --exclude-modules MODULE [MODULE ...] + -em MODULE [MODULE ...], --exclude-modules MODULE [MODULE ...] Exclude these modules. - -f, --flags FLAG [FLAG ...] + -f FLAG [FLAG ...], --flags FLAG [FLAG ...] Enable modules by flag. Choices: active,affiliates,baddns,cloud-enum,code-enum,download,email-enum,iis-shortnames,invasive,loud,passive,portscan,safe,service-enum,slow,social-enum,subdomain-enum,subdomain-hijack,web,web-heavy,web-paramminer,web-screenshots -lf, --list-flags List available flags. - -rf, --require-flags FLAG [FLAG ...] + -rf FLAG [FLAG ...], --require-flags FLAG [FLAG ...] Only enable modules with these flags (e.g. -rf passive) - -ef, --exclude-flags FLAG [FLAG ...] + -ef FLAG [FLAG ...], --exclude-flags FLAG [FLAG ...] Disable modules with these flags. (e.g. -ef loud) Scan: - -n, --name SCAN_NAME Name of scan (default: random) + -n SCAN_NAME, --name SCAN_NAME + Name of scan (default: random) -v, --verbose Be more verbose -d, --debug Enable debugging -S, --silent Be quiet @@ -101,14 +102,15 @@ Scan: --current-preset Show the current preset in YAML format --current-preset-full Show the current preset in its full form, including defaults - -mh, --module-help MODULE + -mh MODULE, --module-help MODULE Show help for a specific module Output: - -o, --output-dir DIR Directory to output scan results - -om, --output-modules MODULE [MODULE ...] + -o DIR, --output-dir DIR + Directory to output scan results + -om MODULE [MODULE ...], --output-modules MODULE [MODULE ...] Add output module(s). Choices: asset_inventory,csv,discord,elastic,emails,json,kafka,mongo,mysql,nats,neo4j,nmap_xml,postgres,rabbitmq,slack,splunk,sqlite,stdout,subdomains,teams,txt,web_parameters,web_report,webhook,websocket,zeromq - -eom, --exclude-output-modules MODULE [MODULE ...] + -eom MODULE [MODULE ...], --exclude-output-modules MODULE [MODULE ...] Exclude output module(s) -lo, --list-output-modules List available output modules @@ -135,15 +137,15 @@ Misc: --proxy HTTP_PROXY Use this proxy for all HTTP requests --no-proxy HOST [HOST ...] Exclude these hosts from proxy (e.g. localhost *.internal.corp 10.0.0.0/8) - -H, --custom-headers CUSTOM_HEADERS [CUSTOM_HEADERS ...] + -H CUSTOM_HEADERS [CUSTOM_HEADERS ...], --custom-headers CUSTOM_HEADERS [CUSTOM_HEADERS ...] List of custom headers as key value pairs (header=value). - -C, --custom-cookies CUSTOM_COOKIES [CUSTOM_COOKIES ...] + -C CUSTOM_COOKIES [CUSTOM_COOKIES ...], --custom-cookies CUSTOM_COOKIES [CUSTOM_COOKIES ...] List of custom cookies as key value pairs (cookie=value). - --custom-yara-rules, -cy CUSTOM_YARA_RULES + --custom-yara-rules CUSTOM_YARA_RULES, -cy CUSTOM_YARA_RULES Add custom yara rules to excavate - --user-agent, -ua USER_AGENT + --user-agent USER_AGENT, -ua USER_AGENT Set the user-agent for all HTTP requests - --user-agent-suffix, -uas SUFFIX + --user-agent-suffix SUFFIX, -uas SUFFIX Suffix to append to the user-agent EXAMPLES From 508b7f5ce2b1085ae2bb1ae8bafe115d00b17a4d Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 19:36:30 -0400 Subject: [PATCH 26/29] Point docs links at Stable instead of Dev README and CONTRIBUTIONS are GitHub-rendered root files with no docs-site version context; mike publishes only versioned deep links (default: Stable), so these must name a version. Stable matches the rest of the README's links. --- CONTRIBUTIONS.md | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTIONS.md b/CONTRIBUTIONS.md index 59d24e3985..b7155f5318 100644 --- a/CONTRIBUTIONS.md +++ b/CONTRIBUTIONS.md @@ -2,4 +2,4 @@ See our full contribution guide at: -**https://www.blacklanternsecurity.com/bbot/Dev/contribution/** +**https://www.blacklanternsecurity.com/bbot/Stable/contribution/** diff --git a/README.md b/README.md index 3f08a21b6d..2beaa88dcc 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ config: <!-- END BBOT SUBDOMAIN-ENUM PRESET EXPANDABLE --> -BBOT consistently finds 20-50% more subdomains than other tools. The bigger the domain, the bigger the difference. To learn how this is possible, see [How It Works](https://www.blacklanternsecurity.com/bbot/Dev/how_it_works/). +BBOT consistently finds 20-50% more subdomains than other tools. The bigger the domain, the bigger the difference. To learn how this is possible, see [How It Works](https://www.blacklanternsecurity.com/bbot/Stable/how_it_works/). ![subdomain-stats-ebay](https://github.com/blacklanternsecurity/bbot/assets/20261699/de3e7f21-6f52-4ac4-8eab-367296cd385f) From 6166dd71905d83e253c2356c8dc6a10862a448eb Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 21:31:59 -0400 Subject: [PATCH 27/29] Migration guide: engine framework removal, redact_secrets, waf_bypass Note bbot.core.engine / BBOTEngineError / CurlError removal and WebError/DNSError reparenting; add redact_secrets and waf_bypass; sync cloudcheck/asndb dep bounds. --- docs/migration/3.0_breaking_changes.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index 528b6d8ab4..a7fae3a761 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -172,6 +172,7 @@ truth. - `http` -- replaces `httpx`; runs through the in-process [blasthttp](https://github.com/blacklanternsecurity/blasthttp) client. - `webbrute` / `webbrute_shortnames` -- ffuf replacements, also via blasthttp. - `bucket_hetzner`, `shodan_enterprise`, `trajan`, `legba`. +- `waf_bypass` -- finds WAF/CDN bypasses by reaching protected content directly via origin IPs (SimHash content matching, with optional ASN-neighbor exploration). - Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq`. - Lightfuzz submodules `esi` and `ssrf`. @@ -437,6 +438,9 @@ Changes: ### Added - `max_mem_percent` -- global ingress throttle when RSS exceeds the threshold. +- `redact_secrets` (default `true`) -- redacts secret values (API keys, tokens) + when a resolved preset is serialized to YAML (e.g. the generated `bbot.yml`). + Set to `false` to include them. - `web.user_agent_suffix` -- appended to the user agent (previously buried as a hidden CLI flag). - `web.http_rate_limit` -- global rps cap across the shared blasthttp client. @@ -513,6 +517,12 @@ architecture was deleted. `request_batch` / `request_custom_batch` / `curl` methods were replaced by `request()`, `request_batch_stream(urls, threads=10, **kwargs)`, and `download()`. +- **Engine framework removed**: the `bbot.core.engine` module (`EngineBase` / + `EngineClient` / `EngineServer`) and the `BBOTEngineError` exception were + deleted along with the subprocess architecture. `WebError` and `DNSError` now + subclass `BBOTError` directly, and `CurlError` was removed. External code + importing `bbot.core.engine`, `BBOTEngineError`, or `CurlError` must be + updated. - The blasthttp dependency line is `blasthttp>=0.9.0`. Modules that previously instantiated their own `httpx.AsyncClient` or built @@ -595,10 +605,10 @@ If your downstream tooling relied on seeing `HTTP_RESPONSE` or `<3.15`. - **Lockstep deps**: - `radixtarget >=4.0.1,<5` (composition pattern, no longer subclassed) - - `cloudcheck >=11.0.0,<12` + - `cloudcheck >=11.1.0,<12` - `blasthttp >=0.9.0` (new) - `blastdns >=1.9.0,<2` (new) - - `asndb >=1.0.4` (new) + - `asndb >=1.1.0` (new) - `zstandard` (new; used by HTTP body spill) - `httpx` **removed** as a runtime dep. From 6d33cb04354059d898ae07151a9bc1eef62e686b Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 22:16:24 -0400 Subject: [PATCH 28/29] Fix event.md cross-link and add webbrute to nav event.md linked to ../../scanning/events (outside docs root); correct to ../scanning/events.md to match other dev pages. Wire the orphaned webbrute deep-dive page into the Modules nav. --- docs/dev/event.md | 2 +- mkdocs.yml | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/dev/event.md b/docs/dev/event.md index 79f0cc7cb5..7bc7400f28 100644 --- a/docs/dev/event.md +++ b/docs/dev/event.md @@ -1,4 +1,4 @@ -This is a developer reference. For a high-level description of BBOT events including a full list of event types, see [Events](../../scanning/events) +This is a developer reference. For a high-level description of BBOT events including a full list of event types, see [Events](../scanning/events.md) ::: bbot.core.event.base.make_event ::: bbot.core.event.base.event_from_json diff --git a/mkdocs.yml b/mkdocs.yml index 40202f8d91..8a535b70d1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,6 +35,7 @@ nav: - Wayback: modules/wayback.md - Custom YARA Rules: modules/custom_yara_rules.md - Lightfuzz: modules/lightfuzz.md + - Webbrute: modules/webbrute.md - Migration: - 2.x → 3.0 Breaking Changes: migration/3.0_breaking_changes.md - Misc: From 591c25a65094f91e5b30f7b0fddc896758e02529 Mon Sep 17 00:00:00 2001 From: liquidsec <paul.mueller08@gmail.com> Date: Mon, 29 Jun 2026 22:24:18 -0400 Subject: [PATCH 29/29] Add internal modules page to nav --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index 8a535b70d1..946fe6f498 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,6 +31,7 @@ nav: - Configuration: scanning/configuration.md - Modules: - List of Modules: modules/list_of_modules.md + - Internal Modules: modules/internal_modules.md - Nuclei: modules/nuclei.md - Wayback: modules/wayback.md - Custom YARA Rules: modules/custom_yara_rules.md