BBOT Server

[TheTechromancer]

I've started work on a backend for ingesting BBOT scans and tracking them over time.

This "BBOT server" will include both a python and REST API useful for ASM, pentest OSINT, bug bounties, and threat intel, that can be easily spun up with a single command, but also scaled to billions of events when needed.

The goal is to keep a history of changes to each asset, like new open ports, vulnerabilities, technologies, etc., and allow easy querying + alerting. Originally I had used SQLite as the default database, with a Postgres option for bigger setups. But this setup is brittle and icky to maintain.

I'm beginning to sense that something like Elastic might be better. Specifically, I like the idea of having a central store of events that can be easily browsed by power users, separate from BBOT server.

@colin-stubbs @nicpenning @CarsonHrusovsky Do you want to show off some of the work you've done with Elastic? I'm curious to see if it would be a good fit as a backend for BBOT server.

[nicpenning]

Hello, Techro!

The dashboards you see in the integration is a good example of what use today. We have customized it a little bit because we do more enrichments to the data with organizational related data (What is known vs unknown assets).
https://github.com/elastic/integrations/blob/main/packages/bbot/img/BBOT-Dashboard.png

Asset management can be tricky so the only way we can have the "current" state is by filtering down to a specific entity like evilcorp.com for example then apply a filter for the last scan date / scan name.

Since Elastic is a document store and not a relational database, we have to get creative with queries to present the information we seek. However, this is more of a limitation of using Kibana (The GUI) for Elasticsearch which is what that dashboard above is from. If you were to index all the data in Elasticsearch, you could use the API calls to develop your own stats/metrics and visualizations (I just don't know how that works).

External tooling would be needed to do diffs with current/previous scans to show a "Latest info" on an asset, which we don't do today, but would be worthwhile.

We can use a "New Terms" alert type rule with Kibana to show data in fields that haven't been seen in 30 days for example. This would allow us to alert/create events on any new findings over time via "Alerts".

Are you thinking about having Elasticsearch included in this BBOT build that would store these many events over time that users of BBOT could query specifically for, or maybe perhaps show in a new BBOT Web App the results of scans or something similar?

FYI - Elasticsearch is going open source again, so that may or may not help your design to embed that type of product into BBOT (if that is what you are looking to do).

[TheTechromancer]

Thanks @nicpenning.

Are you thinking about having Elasticsearch included in this BBOT build that would store these many events over time that users of BBOT could query specifically for, or maybe perhaps show in a new BBOT Web App the results of scans or something similar?

Yes, exactly. The idea would be to make full use of Elastic's features, while adding on a layer of custom functionality that makes it easy to track assets over time, diff between scans, etc.

Basically, I loathe SQL and want a schemaless backend. I'm not sure yet if Elastic is the best option, but I want to try it out and see.

It's been a real challenge to find a backend that makes full use of BBOT's data. In a way, a BBOT scan is NoSQL JSON documents. It's also time-series data, like logs. But then there are complex relationships between the documents. These relationships are why I think a graph database would be best.

However, we've already tried Neo4j, and while it was pretty neat, it was very slow, and its UI was incapable of rendering >1000 nodes. Not production ready.

Basically, I am exploring these types of databases:

• Search Engine (e.g. Elasticsearch)
• Time-series DB (e.g. Prometheus)
• NoSQL (e.g. MongoDB)
• Graph DB (e.g. Dgraph)

Probably none of them will be perfect, but my hope is to find the best possible one, and really commit to it, and use it as a foundation to build a really nice Python/REST API. This will most likely be open-source, and have a CLI to go with it. There won't be an official open-source GUI (because we are selling one, and we have to make money somehow 🫠), but I'm sure the community will make their own GUIs, similar to how @Sh4d0wHunt3rX has done.

[nicpenning]

Okay, thanks for the context. Elasticsearch is not a graph database for sure. The Graph feature was more for Kibana and I don't think that will scale well for visualization anyways. There is a new query language called ES|QL that is a bit more complex but unsure how it fits the use cases. There will still be much work to have the diff capability between BBOT executions since there isn't anything native in the platform.

[TheTechromancer]

Here I'll keep a running tally of the pros and cons of each database, as I explore them. Anyone is free to weigh in.

SQL

✅ PROS:

• Stable
• Allows relationships between tables
• Mature library support (SQLAlchemy)
• Can be extremely lightweight (SQLite)
• Integrates well with FastAPI (SQLModel)
• Plenty of query tools / visualizers for power users

❌ CONS

• Strict schema
  • makes it hard to evolve codebase over time
  • makes maintenance a pain (db migrations 🤮)
  • makes it difficult to add custom fields (i.e. custom server modules)
• Poor indexing of JSON data (i.e. nested fields like tags)

Elasticsearch

✅ PROS:

• Open source, large community
• Flexible Schema
• Full text indexing, powerful search
• Nice query / visualization tools for power users (Kibana)

❌ CONS

• Resource-intensive (Java)
  • Indexes everything by default, maybe overkill
• No builtin graph functionality (except this?)

MongoDB (NoSQL)

✅ PROS:

• Open source, stable
• Schema-less
• Extremely fast and efficient (C++)
• Simple and easy-to-use Python API
• Scalable

❌ CONS

• No query / visualization tools for power users
• No concept of relationships (except this?)

DGraph

✅ PROS:

• Schema-less
• Fast and efficient (Golang)
• Python API
• Native GraphQL
  • Schema can be generated from Pydantic models!
• Evolvable schema without migrations (theoretically)
• Can add/remove indexes at will
• Graph database
• Simple query tool / visualizer for power users (Ratel)

❌ CONS

• TODO

[nicpenning]

@TheTechromancer, if you would like some more information about using Elasticsearch within this project, you can reach out to my trusted contact from Elastic who can provide some guidance on this topic: [email protected] - you can give them my name.

[colin-stubbs]

@TheTechromancer I think this is a great idea, assuming you want to do maintain it rather than support integrations in some way with some of the better products in the asset/vuln management type space.

There's a couple that I think would be worth looking at either way, e.g. RunZero aka. Rumble.Run and Nucleus Security, both for inspiration and consideration as official integrations.

Elastic's "asset" awareness style type functionality, in the commercial licensing variations, would be worth looking at for integration to as well.

I'd love to help when I can... the last few months have been hectic but I'll be settled in one place again from December.

[TheTechromancer]

Hey it'll be nice to have you back; I wondered what you were up to 👀

[TheTechromancer]

As we're spinning up our ASM offering I've got a chance to check out a few similar ones like Palo Alto's Xpanse and Axonius. I've been really curious about RunZero since it's HD Moore's company but haven't got a chance to use it yet. Do you have access to a RunZero or Nucleus? I'd actually love to hop in a call sometime and get your thoughts on features etc.

[colin-stubbs]

Yeap, I've got a freebie 100 asset RunZero tenant... there's probably still a URL or something that we can hit to create a BBOT testing specific freebie tenant.

I can probably get a Nucleus tenant via some of the people I've been in contact with there before.

Axonius is, at least from my experience a few years ago, more a meta-level asset management thing. Definitely relevant to big big big enterprise, but my recollection of pricing, lack of relevant features, and complexity (both technical and purchasing processes) meant that we had to rule it out as a part of our solution offerings. Nucleus was a better fit.

[Luoooio]

@TheTechromancer That's an awesome idea, and I’m really looking forward to seeing such a backend!