diff --git a/AGENTS.md b/AGENTS.md index c86e5cf4e1..4ac4a89725 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -81,6 +81,18 @@ ruff format --check # verify formatting without changes --- +## AI Use Disclosure + +Use of AI is not prohibited -- and in many cases, encouraged. However, when reviewing a PR, it is helpful for the reviewer to know the extent to which AI was used, and which model. +Please add a small section at the bottom of the PR with the header: `### AI Use Disclosure`, followed by the following information: + +* Extent of the AI use. For example, was this fully autonomous by the AI, or was it a collaborative back-and-forth, or did the user just use the AI to review their work, etc. +* Model Used + +This should only apply to external contributors, not members of the blacklanternsecurity organization. + +--- + ## Architecture Overview ### How a Scan Works diff --git a/README.md b/README.md index 192ddbedf2..d4b6496f10 100644 --- a/README.md +++ b/README.md @@ -202,7 +202,7 @@ flags: bbot -t evilcorp.com -p kitchen-sink # roughly equivalent to: -bbot -t evilcorp.com -p subdomain-enum cloud-enum code-enum email-enum spider web paramminer dirbust-light web-screenshots +bbot -t evilcorp.com -p subdomain-enum cloud-enum code-enum email-enum spider web paramminer webbrute web-screenshots ``` @@ -221,7 +221,7 @@ include: - spider - web - paramminer - - dirbust-light + - webbrute - web-screenshots - baddns-heavy @@ -393,12 +393,15 @@ For details, see [Configuration](https://www.blacklanternsecurity.com/bbot/Stabl - [Tips and Tricks](https://www.blacklanternsecurity.com/bbot/Stable/scanning/tips_and_tricks) - [Advanced Usage](https://www.blacklanternsecurity.com/bbot/Stable/scanning/advanced) - [Configuration](https://www.blacklanternsecurity.com/bbot/Stable/scanning/configuration) + - [Scan Sanity](https://www.blacklanternsecurity.com/bbot/Stable/scanning/scan_sanity) - **Modules** - [List of Modules](https://www.blacklanternsecurity.com/bbot/Stable/modules/list_of_modules) + - [Internal Modules](https://www.blacklanternsecurity.com/bbot/Stable/modules/internal_modules) - [Nuclei](https://www.blacklanternsecurity.com/bbot/Stable/modules/nuclei) - [Wayback](https://www.blacklanternsecurity.com/bbot/Stable/modules/wayback) - [Custom YARA Rules](https://www.blacklanternsecurity.com/bbot/Stable/modules/custom_yara_rules) - [Lightfuzz](https://www.blacklanternsecurity.com/bbot/Stable/modules/lightfuzz) + - [Webbrute](https://www.blacklanternsecurity.com/bbot/Stable/modules/webbrute) - **Migration** - [2.x → 3.0 Breaking Changes](https://www.blacklanternsecurity.com/bbot/Stable/migration/3.0_breaking_changes) - **Misc** diff --git a/bbot/presets/kitchen-sink.yml b/bbot/presets/kitchen-sink.yml index e53e902b7d..a9af4b167f 100644 --- a/bbot/presets/kitchen-sink.yml +++ b/bbot/presets/kitchen-sink.yml @@ -8,7 +8,7 @@ include: - spider - web - paramminer - - dirbust-light + - webbrute - web-screenshots - baddns-heavy diff --git a/bbot/presets/nuclei/nuclei-heavy.yml b/bbot/presets/nuclei/nuclei-heavy.yml index d185ecef55..65fa6fc1de 100644 --- a/bbot/presets/nuclei/nuclei-heavy.yml +++ b/bbot/presets/nuclei/nuclei-heavy.yml @@ -22,7 +22,7 @@ conditions: {% endif %} -# Example for also running a dirbust +# Example for also running webbrute #include: -# - dirbust-light +# - webbrute diff --git a/bbot/presets/nuclei/nuclei-technology.yml b/bbot/presets/nuclei/nuclei-technology.yml index fc63767f6d..9cb289e539 100644 --- a/bbot/presets/nuclei/nuclei-technology.yml +++ b/bbot/presets/nuclei/nuclei-technology.yml @@ -17,7 +17,7 @@ conditions: {{ warn("Running nuclei with spider enabled is generally not recommended. Consider removing 'spider' preset.") }} {% endif %} -# Example for also running a dirbust +# Example for also running webbrute #include: -# - dirbust-light +# - webbrute diff --git a/bbot/presets/web/dirbust-heavy.yml b/bbot/presets/web/webbrute-heavy.yml similarity index 100% rename from bbot/presets/web/dirbust-heavy.yml rename to bbot/presets/web/webbrute-heavy.yml diff --git a/bbot/presets/web/dirbust-light.yml b/bbot/presets/web/webbrute.yml similarity index 100% rename from bbot/presets/web/dirbust-light.yml rename to bbot/presets/web/webbrute.yml diff --git a/bbot/test/test_step_1/test_presets.py b/bbot/test/test_step_1/test_presets.py index fa2be51253..d1a4117866 100644 --- a/bbot/test/test_step_1/test_presets.py +++ b/bbot/test/test_step_1/test_presets.py @@ -916,7 +916,7 @@ def test_preset_include(): # can't use both include= and presets= at the same time with pytest.raises(ValueError): - preset = Preset(presets=["subdomain-enum"], include=["dirbust-light"]) + preset = Preset(presets=["subdomain-enum"], include=["webbrute"]) @pytest.mark.asyncio diff --git a/docs/comparison.md b/docs/comparison.md index 183e84319d..239bae9ed2 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -2,7 +2,7 @@ BBOT does a lot more than just subdomain enumeration. However, subdomain enumeration is arguably the most important part of OSINT, and since there's so many subdomain enumeration tools out there, they're the easiest class of tool to compare it to. -Thanks to BBOT's recursive nature (and its `dnsbrute_mutations` module with its NLP-powered subdomain mutations), it typically finds about 20-25% more than other tools such as `Amass` or `theHarvester`. This holds true especially for larger targets like `delta.com` (1000+ subdomains): +Thanks to BBOT's recursive nature (and its `dnsbrute_mutations` module with its NLP-powered subdomain mutations), it typically finds about 20-50% more than other tools such as `Amass` or `theHarvester`. This holds true especially for larger targets like `delta.com` (1000+ subdomains): ### Subdomains Found diff --git a/docs/contribution.md b/docs/contribution.md index 5d4015a3f8..e6f2099d68 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -31,7 +31,7 @@ Modern AI is an incredibly valuable tool for development. We definitely use it. - **No low quality submissions.** If you just submitted the same slop PR to 20 other repos, we are going to close it. If it ignores all the normal patterns in use in BBOT and does its own thing, we are going to close it. If you make a good-faith attempt at doing it right, we're more than happy to help you along. - **Don't let the AI edit tests.** Of course, there will be some times where a test edit is legitimately needed, but this is a common antipattern for AI agents. -We don't require AI disclosure, but it's not discouraged either (usually we can tell though). +We don't require AI disclosure, but it's welcome (and honestly, we can usually tell). If you'd like to include one, add a short **AI Use Disclosure** at the bottom of your PR: how extensively AI was used (fully autonomous, a collaborative back-and-forth, or an AI review of your own work) and which model. We provide an [AGENTS.md](https://github.com/blacklanternsecurity/bbot/blob/dev/AGENTS.md) file in the repo root. Feed this to your LLM before working on BBOT. It describes our conventions and should help steer AI-assisted contributions in the right direction. diff --git a/docs/dev/module_howto.md b/docs/dev/module_howto.md index ec68850544..9554466aeb 100644 --- a/docs/dev/module_howto.md +++ b/docs/dev/module_howto.md @@ -23,7 +23,7 @@ class whois(BaseModule): watched_events = ["DNS_NAME"] # watch for DNS_NAME events produced_events = ["DNS_NAME"] # we produce DNS_NAME events flags = ["passive", "safe"] - meta = {"description": "Query WhoisXMLAPI for related domains"} + meta = {"description": "Query WhoisXMLAPI for related domains", "created_date": "2024-01-01", "author": "@you"} class Config(BaseModuleConfig): api_key: str = Field("", description="WhoisXMLAPI Key", sensitive=True, mandatory=True) diff --git a/docs/migration/3.0_breaking_changes.md b/docs/migration/3.0_breaking_changes.md index a7fae3a761..0e1d965434 100644 --- a/docs/migration/3.0_breaking_changes.md +++ b/docs/migration/3.0_breaking_changes.md @@ -23,6 +23,9 @@ that actually drive passive modules at scan start). | `-s / --silent` | `-S / --silent` (capitalized; `-s` reassigned to `--seeds`) | | `--allow-deadly` | removed (see flag changes below) | +!!! warning "The `-s` flag changed meaning" + In 2.x `-s` was `--silent`; in 3.0 it's `--seeds`, and silent moved to `-S`. A script still passing `-s` for quiet output will now be adding a seed instead, with no error. + Behavior: - If `--seeds` is omitted, seeds default to whatever was passed to `--targets`, @@ -56,20 +59,27 @@ also removed. ## Flags +### Renamed + +| 2.x | 3.0 | +|----------------------|--------------------| +| `web-basic` | `web` | +| `web-thorough` | `web-heavy` | + +### Removed + | Removed | Replacement / notes | |---------------------|---------------------| | `aggressive` | use `loud` (network volume) and/or `invasive` (destructive) | | `deadly` | dropped along with `--allow-deadly`; affected modules now live under regular flags | -| `noisy` | renamed to `loud` | -| `web-basic` | renamed to `web` | -| `web-thorough` | renamed to `web-heavy` | + +### Added | Added | Meaning | |-------------|---------| | `invasive` | Intrusive or potentially destructive | | `safe` | Non-intrusive and non-destructive (now enforced on every module) | | `download` | Modules that download files, apps, or repositories | -| `web-heavy` | More advanced web scanning functionality | Every module must now declare at least one of `passive` / `active` **and** at least one of `safe`, `loud`, or `invasive`. Custom modules carrying the removed @@ -84,22 +94,19 @@ flags will fail validation. | 2.x | 3.0 | |----------------------|--------------------| | `web-basic` | `web` | +| `web-thorough` | `web-heavy` | | `nuclei-intense` | `nuclei-heavy` | | `spider-intense` | `spider-heavy` | - -### Removed - -- `web-thorough` (use `web-heavy`) -- `baddns-intense` (the baddns preset tiers were rebuilt; use `baddns` / - `baddns-heavy`) -- `web/lightfuzz-medium`, `web/lightfuzz-superheavy` (lightfuzz preset tiers - were rebuilt; see `web/lightfuzz`, `web/lightfuzz-heavy`, - `web/lightfuzz-max`) +| `baddns-intense` | `baddns-heavy` | +| `dirbust-light` | `webbrute` | +| `dirbust-heavy` | `webbrute-heavy` | +| `lightfuzz-medium` | `lightfuzz` | +| `lightfuzz-superheavy` | `lightfuzz-max` | ### Added -`baddns`, `baddns-heavy`, `web-heavy`, `web/lightfuzz`, `web/lightfuzz-max`, -`web/paramminer-heavy`. +`baddns`, `waf-bypass`, `wayback`, `wayback-heavy`, `web/paramminer-heavy`, +`web/virtualhost`, `web/virtualhost-heavy`. ### Syntax @@ -127,14 +134,8 @@ gone too; inject secrets with shell expansion ### Removed (no direct replacement) -- `azure_realm` -- `digitorus` -- `passivetotal` -- `sitedossier` - `smuggler` -- `wappalyzer` -- `wpscan` (dropped over a heavy Ruby/compiler install footprint and an - unmaintained upstream) +- `wpscan` ### Removed and replaced @@ -143,11 +144,9 @@ gone too; inject secrets with shell expansion | `httpx` | `http` | | `ffuf` | `webbrute` | | `ffuf_shortnames`| `webbrute_shortnames` | -| `bucket_azure` | `bucket_microsoft` | | `extractous` | `kreuzberg` | | `output.http` | `output.webhook` | | `vhost` | `virtualhost` (rebuilt; emits the new `VIRTUAL_HOST` event) | -| `censys` (single module) | split into `censys_dns` and `censys_ip` | If you used any of these in a custom preset or `-m` / `-em` invocation, update the module name accordingly. The `--list-modules` output is the source of @@ -164,17 +163,23 @@ truth. `output.webhook` in 3.0. Old scans that ran `bbot -om http ...` were emitting events to a webhook; - in 3.0 the equivalent is `bbot -om webhook ...`. The new `bbot -m http ...` - is a scan module that probes URLs, which is a completely different thing. + in 3.0 the equivalent is `bbot -om webhook ...`. By contrast, `bbot -m http + ...` enables the scan module, which probes URLs and is unrelated to the + former output module. + +### Late 2.x removals + +These were dropped during the final 2.x releases, so they're already gone if you're on the latest 2.x, but may still surprise you if you're upgrading from an earlier 2.x version: + +- `digitorus`, `passivetotal`, `sitedossier`, `wappalyzer` (removed, no replacement) +- `azure_realm` (functionality merged into `azure_tenant`) +- `bucket_azure` (now `bucket_microsoft`) +- `censys` (split into `censys_dns` and `censys_ip`) -### New modules worth knowing about +### New modules -- `http` -- replaces `httpx`; runs through the in-process [blasthttp](https://github.com/blacklanternsecurity/blasthttp) client. -- `webbrute` / `webbrute_shortnames` -- ffuf replacements, also via blasthttp. -- `bucket_hetzner`, `shodan_enterprise`, `trajan`, `legba`. -- `waf_bypass` -- finds WAF/CDN bypasses by reaching protected content directly via origin IPs (SimHash content matching, with optional ASN-neighbor exploration). -- Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq`. -- Lightfuzz submodules `esi` and `ssrf`. +- Scan: `bucket_hetzner`, `shodan_enterprise`, `trajan`, `waf_bypass` +- Output: `elastic`, `kafka`, `mongo`, `nats`, `rabbitmq`, `zeromq` ### Module API @@ -286,12 +291,6 @@ rest of the scan and the scan continues without it. This is separate from the ASN-as-target expansion above, which must abort because there would be nothing to scan. -### SCAN event - -The `SCAN` event now includes a `network` key containing the scanner's hostname, -primary outbound IP, network interfaces (IPv4/IPv6 with netmasks), and default -routes. Useful for correlating scan activity back to a specific agent. - ### JSON output: before and after If you have downstream tooling that parses BBOT's NDJSON output, the schema @@ -513,7 +512,8 @@ architecture was deleted. Rust client. The `self.helpers.dns.resolver` `dnspython` resolver is gone. - **HTTP**: `bbot/core/helpers/web/client.py` and `web/engine.py` were removed. `WebHelper` no longer inherits from `EngineClient`. All HTTP goes - through the shared `self.helpers.blasthttp` client. The httpx-based + through the shared [blasthttp](https://github.com/blacklanternsecurity/blasthttp) + client (`self.helpers.blasthttp`). The httpx-based `request_batch` / `request_custom_batch` / `curl` methods were replaced by `request()`, `request_batch_stream(urls, threads=10, **kwargs)`, and `download()`. @@ -596,10 +596,11 @@ If your downstream tooling relied on seeing `HTTP_RESPONSE` or ## Dependencies and tooling - **Build system**: migrated from Poetry to [uv](https://docs.astral.sh/uv/) - + hatchling. `pyproject.toml` is now PEP-621. `poetry.lock` is replaced by - `uv.lock`. Dev install: `uv sync --group dev`. The - `poetry-dynamic-versioning` integration is gone; version is now plain - `3.0.0`. + with a hatchling build backend. `pyproject.toml` now follows PEP 621, and + `uv.lock` replaces `poetry.lock`; install dev dependencies with + `uv sync --group dev`. Versioning is static (`version = "3.0.0"` in + `pyproject.toml`) rather than derived from git tags by + `poetry-dynamic-versioning`. - **License**: changed from `GPL-3.0` to `AGPL-3.0`. - **Python**: minimum bumped from `3.9` to `3.10`. Upper bound is now `<3.15`. @@ -614,52 +615,3 @@ If your downstream tooling relied on seeing `HTTP_RESPONSE` or Custom modules that imported `httpx`, `dns.asyncresolver`, `radixtarget` subclasses, or `bbot.db.sql.models` will need to be ported. - ---- - -## Quick migration checklist - -- [ ] Replace `--whitelist` / `whitelist=` with positional target args; only - reach for `--seeds` / `seeds=` when you genuinely want a seed that isn't - in scope. -- [ ] Replace `-s` with `-S` for silent runs. -- [ ] Drop `--allow-deadly`. -- [ ] Rename module references: `httpx -> http`, `ffuf -> webbrute`, - `bucket_azure -> bucket_microsoft`, `extractous -> kreuzberg`, - `output.http -> output.webhook`, `vhost -> virtualhost`, - `censys -> censys_dns / censys_ip`. Drop any `wpscan` references (removed, - no replacement). -- [ ] Rename preset references: `web-basic -> web`, `web-thorough -> web-heavy`, - `*-intense -> *-heavy`. -- [ ] Rename module flags: `noisy -> loud`, `web-basic -> web`, - `web-thorough -> web-heavy`. Drop `aggressive` / `deadly`. Add `safe` / - `loud` / `invasive` to satisfy the new validation rule. -- [ ] Replace any `VULNERABILITY` emit with a `FINDING` carrying - `severity=...`. -- [ ] Update FINDING severity strings: `INFORMATIONAL -> INFO`, - `MODERATE -> MEDIUM`. Add a `confidence` field from the new allowlist. -- [ ] Stop reading `event.data` for URL events; use `event.url` / - `event.pretty_string`. Drop dependencies on `event.confidence` / - `cumulative_confidence`. -- [ ] Stop mutating `event._resolved_hosts` / `event.dns_children` directly; - use the new `add_resolved_host` / `add_dns_child` helpers. -- [ ] Update config keys: `web.httpx_timeout -> web.http_timeout` / - `web.http_timeout_infrastructure`, - `web.httpx_retries -> web.http_retries`, - `web.ssl_verify -> web.ssl_verify_target` / `web.ssl_verify_infrastructure`. -- [ ] If an in-place upgrade fails with a config validation error pointing at - `~/.config/bbot/bbot.yml` or `secrets.yml`, back up your customizations - and run `bbot --reset-config` / `bbot --reset-secrets` to regenerate from - current defaults. -- [ ] Update imports: `bbot.db.sql.models -> bbot.models.sql`. -- [ ] If you ship a custom module that watches URLs but doesn't want - blasthttp auto-enabled, set `_disable_auto_module_deps = True`. -- [ ] Switch your install/build pipeline from Poetry to uv. -- [ ] If you relied on `-om` to replace default output modules, switch to - `-eom` to exclude the ones you don't want (output modules are now - purely additive). -- [ ] Update any JSON parsers that read `event["data"]` for URL events -- - the key is now `"data_json"` and the value is a dict, not a string. - Use `event["data_json"]["url"]` to get the URL string. -- [ ] If your tooling consumed `HTTP_RESPONSE` or `URL_UNVERIFIED` from - output, remove them from `omit_event_types` in your config. diff --git a/docs/modules/custom_yara_rules.md b/docs/modules/custom_yara_rules.md index 72a9008194..cf6b540fb6 100644 --- a/docs/modules/custom_yara_rules.md +++ b/docs/modules/custom_yara_rules.md @@ -51,7 +51,7 @@ rule find_AAAABBBB_regex { ``` -*Note: YARA uses it's own regex engine that is not a 1:1 match with python regexes. This means many existing regexes will have to be modified before they will work with YARA. The good news is: YARA's regex engine is FAST, immensely more fast than pythons!* +*Note: YARA uses its own regex engine that is not a 1:1 match with Python regexes. This means many existing regexes will have to be modified before they will work with YARA. The good news is: YARA's regex engine is FAST, immensely faster than Python's!* Further discussion of art of writing complex YARA rules goes far beyond the scope of this documentation. A good place to start learning more is the [official YARA documentation](https://yara.readthedocs.io/en/stable/writingrules.html). @@ -74,7 +74,7 @@ Example with no description provided: Example with the description added: ``` -[FINDING] {"description": "Custom Yara Rule [AAAABBBB] with description: [contains our test string] Matched via identifier [str1]", "host": "example.com, "url": "http://example.com"} excavate +[FINDING] {"description": "Custom Yara Rule [AAAABBBB] with description: [contains our test string] Matched via identifier [str1]", "host": "example.com", "url": "http://example.com"} excavate ``` That FINDING was produced with the following signature: @@ -143,6 +143,7 @@ rule ContainsTitle description = "Contains an HTML title tag" severity = "HIGH" confidence = "CONFIRMED" + strings: $title_value = /(.*)?<\/title>/i condition: $title_value diff --git a/docs/modules/internal_modules.md b/docs/modules/internal_modules.md index cb189ae196..aac1126c30 100644 --- a/docs/modules/internal_modules.md +++ b/docs/modules/internal_modules.md @@ -1,4 +1,4 @@ -# List of Modules +# Internal Modules ## What are internal modules? @@ -25,11 +25,11 @@ These modules are executing core functionality that is normally essential for a Summarize statistics at the end of a scan. Disable if you don't want to see this table. -### cloud +### cloudcheck The cloud module looks at events and tries to determine if they are associated with a cloud provider and tags them as such, and can also identify certain cloud resources -### dns +### dnsresolve The DNS internal module controls the basic DNS resolution the BBOT performs, and all of the supporting machinery like wildcard detection, etc. @@ -41,12 +41,11 @@ Here is a summary of the data it produces: #### URLs -By extracting URLs from all visited pages, this is actually already half of a web-spider. The other half is recursion, which is baked in to BBOT from the ground up. Therefore, protections are in place by default in the form of `web_spider_distance` and `web_spider_depth` settings. These settings govern restrictions to URLs recursively harvested from HTTP responses, preventing endless runaway scans. However, in the right situation the controlled use of a web-spider is extremely powerful. +By extracting URLs from all visited pages, this is actually already half of a web-spider. The other half is recursion, which is baked in to BBOT from the ground up. Therefore, protections are in place by default in the form of the `web.spider_distance` and `web.spider_depth` settings. These settings govern restrictions to URLs recursively harvested from HTTP responses, preventing endless runaway scans. However, in the right situation the controlled use of a web-spider is extremely powerful. #### Parameter Extraction -Parameter Extraction -The parameter extraction functionality identifies and extracts key web parameters from HTTP responses, and produced `WEB_PARAMETER` events. This includes parameters found in GET and POST requests, HTML forms, and jQuery requests. Currently, these are only used by the `hunt` module, and by the `paramminer` modules, to a limited degree. However, future functionality will make extensive use of these events. +The parameter extraction functionality identifies and extracts key web parameters from HTTP responses, and produces `WEB_PARAMETER` events. This includes parameters found in GET and POST requests, HTML forms, and jQuery requests. Currently, these are only used by the `hunt` module, and by the `paramminer` modules, to a limited degree. However, future functionality will make extensive use of these events. #### Email Extraction @@ -70,7 +69,7 @@ The non-HTTP scheme detection capability extracts URLs with non-HTTP schemes, su #### Custom Yara Rules -Excavate supports the use of custom YARA rules, which will be added to the other rules before the scan start. For more info, view this. +Excavate supports the use of custom YARA rules, which will be added to the other rules before the scan starts. For more info, see [Custom YARA Rules](custom_yara_rules.md). ### speculate diff --git a/docs/modules/lightfuzz.md b/docs/modules/lightfuzz.md index 6aa7c1bff7..194da52273 100644 --- a/docs/modules/lightfuzz.md +++ b/docs/modules/lightfuzz.md @@ -147,13 +147,13 @@ If the response contains `1787569` (or `1,787,569`), the expression was evaluate Identifies cryptographic parameters and probes for cryptographic vulnerabilities. This submodule has several stages with varying confidence levels. -**Stage 0 — Keystream Reuse / Many-Time-Pad** (HIGH/CONFIRMED to MEDIUM/PROBABLE; zero HTTP requests): A cross-value passive check that runs *before* the entropy gate. When a stream cipher reuses a single keystream across multiple encryptions (no IV, fixed key — a common homebrew mistake), XOR-ing any two ciphertexts yields the XOR of their plaintexts. For natural-language or identifier plaintexts that share a prefix, the result begins with a run of zero bytes — mathematically impossible under correct encryption. +**Stage 0 — Keystream Reuse / Many-Time-Pad** (HIGH/CONFIRMED to MEDIUM/MEDIUM; zero HTTP requests): A cross-value passive check that runs *before* the entropy gate. When a stream cipher reuses a single keystream across multiple encryptions (no IV, fixed key — a common homebrew mistake), XOR-ing any two ciphertexts yields the XOR of their plaintexts. For natural-language or identifier plaintexts that share a prefix, the result begins with a run of zero bytes — mathematically impossible under correct encryption. The detector gathers candidate ciphertexts from the parameter's own value plus every value in `additional_params` (sibling form fields) and `same_param_values` (other observed values for the same parameter across a single page, e.g. sort links with distinct hex-encoded keys). It decodes each via the existing `format_agnostic_decode()` helper, pairwise-XORs them, and scores the result: - **HIGH / CONFIRMED** — a leading run of ≥ 5 zero bytes (two plaintexts share their first 5+ bytes; impossible under correct encryption with fresh IVs) -- **HIGH / PROBABLE** — a 3-4 byte zero run, or ≥ 95% of XOR bytes fall in the `[0x00, 0x60]` "printable-ASCII XOR printable-ASCII" range -- **MEDIUM / PROBABLE** — ≥ 90% ASCII-XOR-ASCII bytes but no zero-run +- **HIGH / HIGH** — a 3-4 byte zero run, or ≥ 95% of XOR bytes fall in the `[0x00, 0x60]` "printable-ASCII XOR printable-ASCII" range +- **MEDIUM / MEDIUM** — ≥ 90% ASCII-XOR-ASCII bytes but no zero-run This stage bypasses the entropy gate (Stage 1) because the short ASCII plaintexts most vulnerable to this misuse often produce ciphertext below the 4.5-bit threshold the gate would otherwise filter out. diff --git a/docs/modules/nuclei.md b/docs/modules/nuclei.md index c51e041067..ee36838145 100644 --- a/docs/modules/nuclei.md +++ b/docs/modules/nuclei.md @@ -72,7 +72,7 @@ This is the default setting, and will use all templates. However, if you're look #### Technology -This is equivalent to the Nuclei '-as' scan option. It only use templates that match detected technologies, using wappalyzer-based signatures. This can be a nice way to run a light-weight scan that still has a chance to find some good vulnerabilities. +This is equivalent to the Nuclei '-as' scan option. It only uses templates that match detected technologies, using wappalyzer-based signatures. This can be a nice way to run a light-weight scan that still has a chance to find some good vulnerabilities. #### Budget @@ -86,7 +86,7 @@ For only the cost of one (or more) "extra" request(s) per host, it can activate The budget parameter is the # of extra requests per host you are willing to send to "feed" Nuclei templates (defaults to 1). For those times when vulnerability scanning isn't the main focus, but you want to look for easy wins. -Of course, there is a rapidly diminishing return when you set he value to more than a handful. Eventually, this becomes 1 template per 1 budget value increase. However, in the 1-10 range there is a lot of value. This graphic should give you a rough visual idea of this concept. +Of course, there is a rapidly diminishing return when you set the value to more than a handful. Eventually, this becomes 1 template per 1 budget value increase. However, in the 1-10 range there is a lot of value. This graphic should give you a rough visual idea of this concept. ![Nuclei Budget Mode](https://github.com/blacklanternsecurity/bbot/assets/24899338/08a3429c-5a73-437b-84de-27c07d85a529) diff --git a/docs/modules/wayback.md b/docs/modules/wayback.md index 972949cb53..e4bb10c774 100644 --- a/docs/modules/wayback.md +++ b/docs/modules/wayback.md @@ -98,7 +98,7 @@ Wayback's extended features are also enabled in several other presets: | Preset | Wayback Config | |-----------------------|-----------------------------------------| | `kitchen-sink` | `urls`, `parameters`, `archive` | -| `dirbust-heavy` | `urls` | +| `webbrute-heavy` | `urls` | | `nuclei-heavy` | `urls` | | `lightfuzz-heavy` | `urls`, `parameters` | | `lightfuzz-max` | `urls`, `parameters`, `archive` | diff --git a/docs/modules/webbrute.md b/docs/modules/webbrute.md index c48faae0d4..a28475b4a6 100644 --- a/docs/modules/webbrute.md +++ b/docs/modules/webbrute.md @@ -18,7 +18,7 @@ bbot -t example.com -m webbrute bbot -t example.com -m webbrute -c modules.webbrute.extensions=php,asp,jsp # Aggressive mode with larger wordlist and recursion -bbot -t example.com -p dirbust-heavy +bbot -t example.com -p webbrute-heavy ``` ## Features @@ -142,7 +142,7 @@ Hits are collected during the response stream but are not emitted immediately. B BBOT includes two presets for directory brute-forcing: -### `dirbust-light` +### `webbrute` Surface-level directory discovery with a smaller wordlist. Good for quick scans. @@ -151,10 +151,10 @@ Surface-level directory discovery with a smaller wordlist. Good for quick scans. - WAF avoidance enabled ```bash -bbot -t example.com -p dirbust-light +bbot -t example.com -p webbrute ``` -### `dirbust-heavy` +### `webbrute-heavy` Aggressive recursive brute-forcing with a large extension list. Includes spidering and wayback URL extraction to discover additional directories to fuzz. WAF avoidance is disabled so brute-forcing proceeds even against WAF-protected hosts. @@ -165,5 +165,5 @@ Aggressive recursive brute-forcing with a large extension list. Includes spideri - `avoid_wafs` disabled ```bash -bbot -t example.com -p dirbust-heavy +bbot -t example.com -p webbrute-heavy ``` diff --git a/docs/scanning/events.md b/docs/scanning/events.md index efc380ef90..749a1d32be 100644 --- a/docs/scanning/events.md +++ b/docs/scanning/events.md @@ -27,7 +27,7 @@ Each BBOT event has the following attributes. Not all of these attributes are vi - `.id`: an identifier representing the event type + a SHA1 hash of its data (note: multiple events can have the same `.id`) - `.uuid`: a universally unique identifier for the event (e.g. `DNS_NAME:6c96d512-090a-47f0-82e4-6860e46aac13`) - `.scope_description`: describes the scope of the event (e.g. `in-scope`, `affiliate`, `distance-2`) -- `.data`: the actual discovered data (for some events like `DNS_NAME` or `IP_ADDRESS`, this is a string. For other more complex events like `HTTP_RESPONSE`, it's a dictionary) +- `.data`: the actual discovered data (for some events like `DNS_NAME` or `IP_ADDRESS`, this is a string. For other more complex events like `HTTP_RESPONSE`, it's a dictionary). In JSON output it's serialized as `data` or `data_json` depending on its type (see below). - `.host`: the hostname or IP address (e.g. `evilcorp.com` or `1.2.3.4`) - `.port`: the port number (e.g. `80`, `443`) - `.netloc`: the network location, including both the hostname and port (e.g. `www.evilcorp.com:443`) @@ -110,6 +110,15 @@ These attributes allow us to construct a visual graph of events (e.g. in [Neo4j] } ``` +### `data` vs `data_json` + +When events are serialized to JSON (`output.json`, the `json` output module, or `--json`), the data field is named after its type, so a given key always holds a consistent type. This makes parsing far easier for scripts and aggregation tools like Elasticsearch: + +- **`data`** -- a string, for events whose data is a string (`DNS_NAME`, `IP_ADDRESS`, `URL`, ...). The example above is a `DNS_NAME`, so its value lives under `data`. +- **`data_json`** -- a dictionary, for events whose data is structured (`HTTP_RESPONSE`, `FINDING`, `STORAGE_BUCKET`, ...). + +Exactly one of the two is present per event. When reading events back, check `data_json` first and fall back to `data`. + For a more detailed description of BBOT events, see [Developer Documentation - Event](../dev/event.md). Below is a full list of event types along with which modules produce/consume them. diff --git a/docs/scanning/output.md b/docs/scanning/output.md index 8715698ed3..b8d6489495 100644 --- a/docs/scanning/output.md +++ b/docs/scanning/output.md @@ -104,6 +104,8 @@ You will then see [events](events.md) like this: } ``` +String-typed events use the `data` key (as above); structured events like `HTTP_RESPONSE` and `FINDING` use `data_json` (a dict) instead. See [Events](events.md) for details. + You can filter on the JSON output with `jq`: ```bash @@ -413,22 +415,79 @@ The `web_report` output module generates a markdown report of web assets, includ The `nmap_xml` output module exports open ports, DNS names, IP addresses, and protocols in Nmap XML format for compatibility with tools that consume Nmap output. -### Message Queues +### Kafka -BBOT supports streaming events as JSON to several message queue systems: +The `kafka` output module publishes events as JSON to a Kafka topic. -| Module | Description | Key Config | -|--------|-------------|------------| -| `kafka` | Publish to a Kafka topic | `bootstrap_servers`, `topic` | -| `rabbitmq` | Publish to a RabbitMQ queue | `url`, `queue` | -| `nats` | Publish to a NATS subject | `servers`, `subject` | -| `zeromq` | Publish to a ZeroMQ PUB socket | `zmq_address` | -| `websocket` | Stream to a WebSocket endpoint | `url`, `token` | +```yaml title="kafka_preset.yml" +output_modules: + - kafka -These can be enabled like any other output module: +config: + modules: + kafka: + bootstrap_servers: localhost:9092 + topic: bbot_events +``` -```bash -bbot -t evilcorp.com -om kafka -c modules.kafka.bootstrap_servers=localhost:9092 modules.kafka.topic=bbot_events +### RabbitMQ + +The `rabbitmq` output module publishes events as JSON to a RabbitMQ queue. + +```yaml title="rabbitmq_preset.yml" +output_modules: + - rabbitmq + +config: + modules: + rabbitmq: + url: amqp://guest:guest@localhost/ + queue: bbot_events +``` + +### NATS + +The `nats` output module publishes events as JSON to a NATS subject. + +```yaml title="nats_preset.yml" +output_modules: + - nats + +config: + modules: + nats: + servers: + - nats://localhost:4222 + subject: bbot_events +``` + +### ZeroMQ + +The `zeromq` output module publishes events as JSON to a ZeroMQ PUB socket. + +```yaml title="zeromq_preset.yml" +output_modules: + - zeromq + +config: + modules: + zeromq: + zmq_address: tcp://localhost:5555 +``` + +### WebSocket + +The `websocket` output module streams events as JSON to a WebSocket endpoint. Set `token` to send an `Authorization: Bearer` header. + +```yaml title="websocket_preset.yml" +output_modules: + - websocket + +config: + modules: + websocket: + url: ws://localhost:8080 + token: my-auth-token ``` ### MongoDB diff --git a/docs/scanning/presets.md b/docs/scanning/presets.md index 87068f4a7e..bdd6fed313 100644 --- a/docs/scanning/presets.md +++ b/docs/scanning/presets.md @@ -142,7 +142,7 @@ Here is the full list of supported top-level keys in a preset YAML file: | `seeds` | list | Seed events to feed into modules. If omitted, targets are used as seeds | | `blacklist` | list | Excluded targets. Takes ultimate precedence | | `modules` | list | Scan modules to enable | -| `output_modules` | list | Output modules (default: `csv`, `human`, `json`) | +| `output_modules` | list | Output modules (default: `csv`, `txt`, `json`) | | `exclude_modules` | list | Modules to exclude | | `flags` | list | Enable all modules with these flags | | `require_flags` | list | Only enable modules that have these flags | diff --git a/docs/scanning/presets_list.md b/docs/scanning/presets_list.md index 419a99a808..9730d99381 100644 --- a/docs/scanning/presets_list.md +++ b/docs/scanning/presets_list.md @@ -90,83 +90,6 @@ Enumerate Git repositories, Docker images, etc. -Modules: [0]("") - -## **dirbust-heavy** - -Recursive web directory brute-force (aggressive) - -??? note "`dirbust-heavy.yml`" - ```yaml title="~/.bbot/presets/web/dirbust-heavy.yml" - description: Recursive web directory brute-force (aggressive) - - include: - - spider - - flags: - - iis-shortnames - - modules: - - webbrute - - wayback - - config: - modules: - iis_shortnames: - # we exploit the shortnames vulnerability to produce URL_HINTs which are consumed by webbrute_shortnames - detect_only: False - webbrute: - avoid_wafs: False - max_depth: 3 - lines: 5000 - extensions: - - php - - asp - - aspx - - ashx - - asmx - - jsp - - jspx - - cfm - - zip - - conf - - config - - xml - - json - - yml - - yaml - # emit URLs from wayback - wayback: - urls: True - ``` - -Category: web - -Modules: [0]("") - -## **dirbust-light** - -Basic web directory brute-force (surface-level directories only) - -??? note "`dirbust-light.yml`" - ```yaml title="~/.bbot/presets/web/dirbust-light.yml" - description: Basic web directory brute-force (surface-level directories only) - - include: - - iis-shortnames - - modules: - - webbrute - - config: - modules: - webbrute: - # wordlist size = 1000 - lines: 1000 - ``` - -Category: web - Modules: [0]("") ## **dotnet-audit** @@ -292,7 +215,7 @@ Everything everywhere all at once - spider - web - paramminer - - dirbust-light + - webbrute - web-screenshots - baddns-heavy @@ -582,10 +505,10 @@ Run nuclei scans against all discovered targets, allowing for spidering, against {% endif %} - # Example for also running a dirbust + # Example for also running webbrute #include: - # - dirbust-light + # - webbrute ``` Category: nuclei @@ -617,10 +540,10 @@ Run nuclei scans against all discovered targets, running templates which match d {{ warn("Running nuclei with spider enabled is generally not recommended. Consider removing 'spider' preset.") }} {% endif %} - # Example for also running a dirbust + # Example for also running webbrute #include: - # - dirbust-light + # - webbrute ``` Category: nuclei @@ -975,6 +898,83 @@ Take screenshots of webpages +Modules: [0]("") + +## **webbrute** + +Basic web directory brute-force (surface-level directories only) + +??? note "`webbrute.yml`" + ```yaml title="~/.bbot/presets/web/webbrute.yml" + description: Basic web directory brute-force (surface-level directories only) + + include: + - iis-shortnames + + modules: + - webbrute + + config: + modules: + webbrute: + # wordlist size = 1000 + lines: 1000 + ``` + +Category: web + +Modules: [0]("") + +## **webbrute-heavy** + +Recursive web directory brute-force (aggressive) + +??? note "`webbrute-heavy.yml`" + ```yaml title="~/.bbot/presets/web/webbrute-heavy.yml" + description: Recursive web directory brute-force (aggressive) + + include: + - spider + + flags: + - iis-shortnames + + modules: + - webbrute + - wayback + + config: + modules: + iis_shortnames: + # we exploit the shortnames vulnerability to produce URL_HINTs which are consumed by webbrute_shortnames + detect_only: False + webbrute: + avoid_wafs: False + max_depth: 3 + lines: 5000 + extensions: + - php + - asp + - aspx + - ashx + - asmx + - jsp + - jspx + - cfm + - zip + - conf + - config + - xml + - json + - yml + - yaml + # emit URLs from wayback + wayback: + urls: True + ``` + +Category: web + Modules: [0]("") <!-- END BBOT PRESET YAML --> @@ -989,8 +989,6 @@ Here is a the same data, but in a table: | baddns-heavy | | Run all baddns modules and submodules. | 3 | baddns, baddns_direct, baddns_zone | | cloud-enum | | Enumerate cloud resources such as storage buckets, etc. | 0 | | | code-enum | | Enumerate Git repositories, Docker images, etc. | 0 | | -| dirbust-heavy | web | Recursive web directory brute-force (aggressive) | 3 | http, wayback, webbrute | -| dirbust-light | web | Basic web directory brute-force (surface-level directories only) | 1 | webbrute | | dotnet-audit | web | Comprehensive scan for all IIS/.NET specific modules and module settings | 8 | ajaxpro, aspnet_bin_exposure, badsecrets, dotnetnuke, http, telerik, webbrute, webbrute_shortnames | | email-enum | | Enumerate email addresses from APIs, web crawling, etc. | 0 | | | fast | | Scan only the provided targets as fast as possible - no extra discovery | 0 | | @@ -1019,4 +1017,6 @@ Here is a the same data, but in a table: | web | | Quick web scan | 0 | | | web-heavy | | Aggressive web scan | 0 | | | web-screenshots | | Take screenshots of webpages | 0 | | +| webbrute | web | Basic web directory brute-force (surface-level directories only) | 1 | webbrute | +| webbrute-heavy | web | Recursive web directory brute-force (aggressive) | 3 | http, wayback, webbrute | <!-- END BBOT PRESETS --> diff --git a/docs/scanning/scan_sanity.md b/docs/scanning/scan_sanity.md new file mode 100644 index 0000000000..46e5e1fffc --- /dev/null +++ b/docs/scanning/scan_sanity.md @@ -0,0 +1,78 @@ +# Scan Sanity Guide + +BBOT's recursive nature is a double-edged sword. Let's talk about the *bad* edge. Any scan is a few settings away from scanning the entire internet, or sending millions of packets to your targets and never finishing. + +There are two things to consider when trying to keep your scan "sane": your scan settings (modules and config choices) and your target. + +## Scan Settings + +Back in the earliest days of BBOT you could run `bbot -m all` to turn on every module. That was fine when there were only a handful of modules, but as BBOT grew it quickly became a terrible idea, so we removed it. + +Imagine you turn on -f `subdomain-enum`, -m `http` and `webbrute`, and -p `lightfuzz`. This *might* be a reasonable scan if you only had a few targets. But let's say your subdomain-enum ends up finding 1000 websites. + +For each of those 1000 websites, webbrute is going to brute-force every directory it knows about. By default that's 5000 requests. With recursion, every directory it finds kicks off a fresh 5000. Then, for every URL that turns up, lightfuzz sends its full suite of probes (in the hundreds). + +But that's not all, because while that's happening the content of each page is being parsed for additional links. If you also turned on `-p spider`, every one of those gets re-emitted, feeding into itself. + +Let's imagine you also enabled `-p wayback-heavy`, so each host also gets a queue of every wayback URL result. This can very quickly devolve into gigantic queues that never drain. And although we've put a lot of effort into optimizing memory, BBOT is memory-unbounded: theoretically you could have a scan big enough that it eventually takes all of your RAM... or more likely, just never finishes. + +On the other hand, with maybe five hard-coded URLs? This might be a pretty nice scan for doing an initial recon before a web assessment. + +We realize that when most people crack open a security tool, they just want to run it with all the settings. That's one of the reasons we added presets in the first place: to help people pick pre-configured settings to suit different tasks, and also to empower people to create their own. But you can stack multiple presets, and things can very quickly get out of hand. + +We allow you a ton of flexibility with your scans: you can add any combination of modules, flags, and presets, add presets while excluding modules, exclude flags, and so on. However, there are a few things you **can** do that are just plain a terrible idea, and might not be obvious. + +### The `active` flag + +!!! danger "Don't select by activity flag" + ```bash + bbot -f active + ``` + +This is bad for multiple reasons! Every module is required to have EITHER `active` or `passive`. So by just selecting `active`, you are guaranteeing that ALL of the most aggressive modules are also included, since they would obviously be active. Furthermore, this WON'T run the passive modules, which are actually pretty important for most any scan! + +This is probably a design flaw we need to work on in the future (perhaps active or passive should be a module attribute, rather than a flag), but for now we're hoping this guide can steer people in the right direction. Where the `active` flag is genuinely useful is with `-ef active`: excluding it when you want to be really careful about how aggressive the scan is. + +If you truly just want the "throw everything at it" experience, we do have a preset for you: `kitchen-sink`. It's basically the most insane scan setting we can officially endorse. + +### Settings best left alone + +!!! warning "Leave these at their defaults unless you know why" + Most of BBOT's config is safe to tune, but a small handful are not. Get one of these wrong and you could easily end up scanning the entire internet or knocking over your DNS resolver. + +- **`scope.search_distance`** (default `0`) is the big one. It controls how many hops out from your target BBOT is allowed to explore. At `0`, modules only act on in-scope events. Raise it and a single certificate full of affiliate names pulls in those hosts, then *their* subdomains, then *their* infrastructure, and scope balloons outward hop by hop until there's no natural stopping point. The config comment says it outright: don't change this unless you know what you're doing. (This is *not* the same as `dns.search_distance`, which only governs DNS resolution and is safer to adjust.) +- **`web.spider_distance`** / **`web.spider_depth`** (default `0` / `1`). A `spider_distance` of `0` means no spidering at all. Raise it and the crawler follows links from links from links, and every URL it turns up feeds webbrute and lightfuzz. On a large site that loop is effectively unbounded. +- **`dns.threads`** / **`dns.brute_threads`** (default `10` / `1000`) control DNS concurrency. Cranking them won't make your scan meaningfully faster (the real lever is adding more resolvers to `/etc/resolv.conf`); it'll just flood your nameservers and get you rate-limited. +- **`dns.runaway_limit`** (default `5`) and **`dns.wildcard_disable`** (default `false`) are guardrails. `runaway_limit` stops BBOT from chasing a malicious chain of DNS records forever, and wildcard detection is the only thing keeping a `*.example.com` wildcard from spawning infinite bogus subdomains. Turning either off removes a safety net that's there for a reason. + +## Scan Target + +When it comes to the scan target, the main thing to keep in mind is that by default, every domain added to a target also brings all of its possible subdomains into scope. Some domains are truly massive, with thousands and thousands of subdomains, and you're saying they are ALL in scope. `--strict-scope` can be useful here: it changes that behavior so subdomains of targets aren't automatically included in scope. Great if you actually just want to target `evilcorp.com` directly, and NOT `a.evilcorp.com`, `b.evilcorp.com`, `c.evilcorp.com`, etc., which would likely be found during the scan. + +Using an IP-based scope is often a great choice, because it allows you to be very specific about what you want scanned when you don't know the full extent of what `*.whatever.com` actually has in it. For example, a lot of big organizations own their own ASN, where they host a lot of content. But they'll also have a lot of services running with third parties, often pointed to by subdomains in `*.whatever.com`. So if you just want targets within `whatever.com`'s ASN but still want to do subdomain discovery, you can use a seed. + +!!! tip "Scope by IP, but still enumerate subdomains" + ```bash + bbot -s whatever.com -t 192.168.0.0/24 + ``` + +Here, BBOT will do subdomain enumeration against whatever.com, but only things that resolve within 192.168.0.0/24 will be in-scope, so none of your active modules will actually go after anything outside that range (a generalization: active modules CAN be configured to step out of scope, but generally aren't). + +## Module Choices + +### Invasive and Loud + +Modules that produce a lot of traffic are flagged with the `loud` flag. Be cautious when using a bunch of these at once. + +Modules that potentially perform invasive actions are flagged with the `invasive` flag. These actions range from mildly rude (submitting registration forms) all the way to executing code on the server. + +### Explosive combinations + +Although there's no flag for this, it's good to be aware of how modules will feed into each other, and consider the recursive effect this will have. Pay attention to a module's `watched_events` and `produced_events`. A few examples: + +`lightfuzz` watches `WEB_PARAMETER` events. `paramminer_*` modules produce them. That's desirable in some circumstances, but the feedback loop can get pretty large on sizable scans. + +`wayback` watches `URL` events (when its more advanced features are enabled) AND produces `URL_UNVERIFIED` events (which become URL events). `webbrute` also watches `URL` events and produces new `URL_UNVERIFIED` events. They both feed each other! This CAN find really obscure endpoints, but when wayback gives you junk (which happens frequently), you're just exponentially amplifying it. + +On the DNS side, `dnsbrute` and `dnsbrute_mutations` are both extremely valuable, and will often find very obscure domain names that won't be discovered any other way. This is why they are included in `subdomain-enum`. However, they are extremely heavy and will add significant time to your scan, running thousands of resolutions for every domain discovered. If you need a faster scan and don't necessarily need that extra 5-10% of discovery, disabling them will make a huge difference. + diff --git a/docs/scanning/tips_and_tricks.md b/docs/scanning/tips_and_tricks.md index ad6d6a05b3..4f8ff0071c 100644 --- a/docs/scanning/tips_and_tricks.md +++ b/docs/scanning/tips_and_tricks.md @@ -47,7 +47,7 @@ bbot -t evilcorp.com -f subdomain-enum -c dns.brute_threads=5000 ### Speed Up Scans with More DNS Resolvers -By far the most effective way to speed up a BBOT scan is to **add more resolvers to `/etc/resolv.conf`**. BBOT's DNS resolver (blastdns) spins up multiple threads per resolver (default: `5`, configurable via `dns.threads`), so more resolvers = more parallelism = faster scans. +By far the most effective way to speed up a BBOT scan is to **add more resolvers to `/etc/resolv.conf`**. BBOT's DNS resolver (blastdns) spins up multiple threads per resolver (default: `10`, configurable via `dns.threads`), so more resolvers = more parallelism = faster scans. For OSINT, it's critical that every resolver is **unfiltered**. Specialized resolvers that try to block ads, malicious domains, etc. will intentionally omit results. Below is a sample `/etc/resolv.conf` with 11 unfiltered public resolvers: @@ -55,7 +55,7 @@ For OSINT, it's critical that every resolver is **unfiltered**. Specialized reso --8<-- "docs/data/resolv-sample.conf" ``` -Copy this to `/etc/resolv.conf` (or append the `nameserver` lines to your existing config). With all 11 resolvers at the default 5 threads each, blastdns will run 55 workers in parallel instead of the typical 5-15 you get from a default OS config. +Copy this to `/etc/resolv.conf` (or append the `nameserver` lines to your existing config). With all 11 resolvers at the default 10 threads each, blastdns will run 110 workers in parallel instead of the typical 10-30 you get from a default OS config. !!! tip If your system uses `systemd-resolved` or `resolvconf`, you may need to configure the upstream forwarders there instead of editing `/etc/resolv.conf` directly. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 502fe0dc1b..2e620514bd 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -37,6 +37,17 @@ bbot --help ## Regenerate Config As a troubleshooting step it is sometimes useful to clear out your older configs and let BBOT generate new ones. This will ensure that new defaults are properly restored, etc. + +BBOT can do this for you (it backs up the originals first): + +```bash +# regenerate config and/or secrets from current defaults +bbot --reset-config --yes +bbot --reset-secrets --yes +``` + +Or do it manually: + ```bash # make a backup of the old configs mv ~/.config/bbot ~/.config/bbot.bak diff --git a/mkdocs.yml b/mkdocs.yml index 946fe6f498..2779d871b2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -29,6 +29,7 @@ nav: - Tips and Tricks: scanning/tips_and_tricks.md - Advanced Usage: scanning/advanced.md - Configuration: scanning/configuration.md + - Scan Sanity: scanning/scan_sanity.md - Modules: - List of Modules: modules/list_of_modules.md - Internal Modules: modules/internal_modules.md