Standardize "data" field to be an object throughout bbot output

[nicpenning]

It seems that today when you run bbot with the output.json (soon to be output.ndjson? :) ) enabled, there is a discrepancy in the data output where some new line delimited json lines contain the data field with just a string but in some cases (such as findings and asn), it will be populated as an object.

This can be an issue when ingesting into some SIEMs. I recommend (and will try to fix) this behavior by starting with this line of code:

bbot/bbot/modules/hunt.py

Line 291 in 97b2c31

data["url"] = url

I would look at transforming this into:

data = {"url" : url}

So that the data will be changed from:

"data" : "evilcorp.com"
To
"data" : "url" : {"evilcorp.com"}

Thus making the data field an object throughout the output.

This is just an example as I don't know if this will fix it.

Another way to do this is making findings and other relevant features that ate making data an object is giving them their own field name outside of data not to conflict with the object vs non-object json mapping inconsistencies.

Thoughts?

[TheTechromancer]

I agree it can be slightly annoying to have to deal with both strings and dictionaries in the event data.

We do already have several internal attributes that are designed to address this, such as .pretty_string which is a human-friendly summary of the data. (I am hard at work on some developer documentation that will cover all these).

As of right now, I don't think we're including them in the JSON output. But we certainly could. I'm open to ideas on how to make it easier to ingest.

Another way to do this is making findings and other relevant features that ate making data an object is giving them their own field name outside of data not to conflict with the object vs non-object json mapping inconsistencies.

I like where you're going with this. Maybe we could have a .data attribute that's guaranteed to be a string and a .data_json attribute that's guaranteed to be a dictionary.

[nicpenning]

I would be happy to try some of these once I can get the dev environment working.

The current state of the output.json file in most circumstances is not usable even findings exist because of the string/dictionary (python language?) or string/ object.

[TheTechromancer]

@kerrymilan @SudoSuBrew @SpamFaux @aconite33 I'd be curious to know any preferences/best-practices you might have for making JSON output easily ingestible by splunk, elastic, etc.

Right now BBOT's JSON output is full of newline-delimited events like this:

{
  "type": "DNS_NAME",
  "data": "blacklanternsecurity.com",
  ...
}

The issue is that the data field is sometimes a string (like above) but sometimes a dictionary/object like this:

{
  "type": "HTTP_RESPONSE",
  "data": {
    "url": "http://blacklanternsecurity.com:80",
    "timestamp": "2023-09-17T07:11:31.229619186-04:00",
    "port": "80",
    "path": "/",
  },
}

So we're looking for the best way to make this ideal for SEIM ingestion while also keeping it as human-readable as possible.

[nicpenning]

Fixing this will make the data ingestable in its rawest state at the very least. I am sure I will have more input once that is done. Curious also to what the crew thinks 🤔

[TheTechromancer]

Once we've figured this out it might also be fun to make a splunk/elastic output module that inserts the data directly in real-time. That's been on my todo list for a while but I haven't had time to get to it.

[nicpenning]

I have some ideas on how to do this. But agreed, let's get the data cleaned up.

[TheTechromancer]

This has been added in #1049.

This PR adds an option to the json output module, siem_friendly, which when set to true, will format the JSON differently:

{
  "type": "DNS_NAME",
  "data": {
    "DNS_NAME": "blacklanternsecurity.com"
  }
}

[nicpenning]

Will this output be supported for the http output module that does JSON as well?

[TheTechromancer]

Sure, it would be easy to add. Can you make an issue so I don't forget?

[nicpenning]

Yes!

[nicpenning]

#1097