bbot: Recursive Internet Scanner and OSINT Automation Framework

Recon Pipelines

Pentesting workflows typically involve a series of steps: Amass or subfinder is used for subdomains, resolver confirms live hosts, Nmap scans live IPs, gowitness takes screenshots of HTTP services, and manual review of screenshots follows. Each step hands off to the next, with each tool producing a file. The pipeline works if you drive it.

bbot changes this process. You point it at a domain, choose your modules, and the scan runs. Subdomains get port-scanned, live web services get crawled, screenshotted, and secrets in web content get flagged. Automation replaces some manual legwork. You can use bbot with existing tools or swap some out, such as Amass, subfinder, resolver, Nmap, and gowitness.

What bbot Is

bbot is an open source recursive internet scanner and OSINT automation framework built by Black Lantern Security. It features hundreds of modules, including subdomain enumeration, port scanning, web crawling, screenshot capture, secret detection, email harvesting, DNS brute-forcing, certificate transparency log queries. Integrations with Shodan, VirusTotal, SecurityTrails, Censys.

Most of these capabilities exist elsewhere; the architecture of bbot stands out. Recursion is key. bbot discovers a subdomain, then scans it. Every applicable module runs against the new target, performing DNS resolution, port scanning, web crawling, screenshot capture, and subdomain enumeration.

The input domain seeds a scan graph, which then expands. The graph stops expanding when the recursion depth is hit or modules exhaust results. Targets can be domains, IPs, or CIDR ranges. Scan configuration lives in YAML files as reusable profiles, which can be defined once and invoked by name. Profiles can be passive, active, or in-scope-only, and it's easy to switch between them.

Module Ecosystem and Scan Scope

Subdomain Enumeration in bbot

bbot runs multiple discovery methods at once. DNS brute-forcing checks a wordlist. crt.sh and Censys are queried for every certificate issued against the target domain. Passive DNS aggregators contribute historical resolution data. Shodan, VirusTotal, and SecurityTrails add their subdomain data. All these methods run in parallel. Results feed the recursive scan graph.

Beyond Subdomain Discovery

The active module set extends the scope. Nmap integration scans resolved hosts for open ports. Web crawling and screenshot capture modules fingerprint live web services. Secret detection modules scan for exposed API keys, credentials, GitHub tokens. Vulnerability detection modules check for common web exposures. Email harvesting is also part of the passive OSINT capability, which includes subdomain discovery, data aggregation.

Module Presets

Module presets simplify engagement setup. The subdomain-enum preset enables passive enumeration, no active probing. This preset is useful for recon before active testing. The web-basic preset adds live host fingerprinting, web service mapping. The kitchen-sink preset enables everything: subdomain-enum, web-basic, and more. Often, a single preset flag replaces thirty individual module selections.

The Recursive Scan Model

Recursion in Scanning

Recursion changes how scans expand. Start with a domain, find subdomains, and then scan those subdomains. Open ports on those hosts get crawled, and crawled content gets checked for secrets, with no extra setup needed.

Scan depth controls recursion. If you set it to 1, only the initial domain's subdomains get scanned. Increase the depth, and subdomains become targets. For most scans, a depth of 1 or 2 works. Big organizations with complex subdomain structures might need more. Deeper scans take longer, and output grows.

Output Structure

Results show the scan graph. Each asset has the module that found it, its parent domain, and a timestamp.

This approach provides more value than a flat subdomain list. You can see which subdomains share infrastructure, how the scope grew from the seed domain, and which methods found each subdomain, such as certificate transparency, DNS brute-forcing, etc.

Configuration, Output, and Integration

YAML configuration makes bbot practical across multiple engagements. Define API keys for external data sources once in a config file. Create scan profiles for different engagement types, such as passive-only reconnaissance, comprehensive active scanning, and web-focused assessment. Invoke a profile by name rather than reconstructing command-line arguments from memory or notes.

API key configuration follows the same model as other multi-source tools and has a practical implication: output depth scales with configured integrations. Subdomain coverage from Shodan, VirusTotal, and SecurityTrails, combined with bbot's native enumeration methods, substantially exceeds what any single source produces. Configure the keys before the first real engagement run.

Output formats include JSON for downstream processing, CSV for structured review. A Neo4j graph database option persists the full scan graph with relationship metadata. The Neo4j integration exposes the scan graph in a queryable form. Queries can be made for all subdomains discovered via certificate transparency versus DNS brute-forcing, or all hosts with specific open port combinations, or all assets where secrets were detected, using the Neo4j instance.

bbot vs Amass: Choosing the Right Tool

Comparison of Amass and bbot

Amass excels at subdomain enumeration. Its graph database tracks changes over time. You run it repeatedly, and it builds a picture of what's new and what's changed. This longitudinal approach is perfect for monthly or quarterly assessments where detecting changes is key.

Amass's unique feature, amass intel, discovers ASN and CIDR info from org names. bbot doesn't match this. For teams monitoring assets continuously, Amass's change detection is a game-changer.

bbot offers a broader range of features. It scans ports, crawls web apps, captures screenshots, finds secrets, and scans for vulnerabilities. These features automate post-enum workflows. With Amass, you would pipe results to Nmap, then gowitness. bbot does it all in one go, with port scanning, web crawling, screenshot capture, secret detection, and vulnerability scanning.

Most teams use both tools. They use Amass for tracking changes over time. They use bbot for a deep dive into the attack surface in one engagement. Amass provides change detection capabilities, subdomain enumeration, and intel discovery. bbot provides vulnerability scanning, port scanning, web crawling, screenshot capture, and secret detection.

Practical Workflow and Limitations

Engagement Workflow

Start with subdomain-enum in passive mode. This establishes your asset inventory without active probing. Review what bbot finds against your engagement rules. Then, move to active modules: port scanning, web crawling, secret detection.

Scope control is key. Large targets with many subdomains and deep recursion can run for hours and produce a lot of data. Set tight target restrictions, such as specific IP ranges or domain patterns. Keep recursion depth low. Automated scans can easily get out of hand.

Secret Detection

The secret detection modules are worth a look. No extra tooling is needed. They scan web content for API keys, credentials, GitHub tokens, AWS keys, and similar material. Developers often leave these on public web interfaces. Automated detection surfaces findings you'd miss in a manual review.

Verdict

bbot is an automated open source attack surface mapping framework. It runs a single configured scan to feed discovery into fingerprinting into secret detection. No manual handoffs or aggregating output files are required.

The tool offers a range of features, including subdomain enumeration, port scanning, web crawling, screenshot capture, and secret detection. You configure it once in YAML, then you're set. This approach beats running several tools per engagement.

Amass offers longitudinal change detection, which bbot does not. If tracking external assets over time is your main need, Amass is a better fit. For a one-off comprehensive attack surface map with lots of automation, bbot is the better choice.

bbot is best suited for pentesters, red teams doing pre-engagement attack surface mapping, and security teams automating external reconnaissance. The project is available on GitHub at blacklanternsecurity/bbot and has documentation at docs.bbot.dev.