Option to Raise FILESYSTEM and WEBSCREENSHOT with base64 blobs #1438
Conversation
|
FYI - Needs work and is not currently functioning or well thought out. |
|
It's a good start. Tomorrow I'll see if I can pitch in and get it working. |
|
@nicpenning are you able to allow edits from maintainers?
|
|
Looks like I already had that set.
|
|
Okay, I made a few changes:
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## stable #1438 +/- ##
======================================
- Coverage 92% 92% -0%
======================================
Files 321 321
Lines 20685 20749 +64
======================================
+ Hits 19014 19057 +43
- Misses 1671 1692 +21 ☔ View full report in Codecov by Sentry. |
|
Phenomenal! |
TheTechromancer
changed the title
|
Please let me know how I can test with custom/flag options, and I can run do some trial runs to see how it shakes out. |
@nicpenning you can test like this: bbot -t evilcorp.com -c file_blobs=true folder_blobs=trueThis will automatically attach binary blobs to any |
| codebase_event = self.make_event( | ||
| {"path": str(repo_path)}, "FILESYSTEM", tags=["git", "folder"], source=event | ||
| ) | ||
| codebase_event = self.make_event({"path": str(repo_path)}, "FILESYSTEM", tags=["git"], source=event) |
There was a problem hiding this comment.
The removal of the folder tag here will stop the unstructured module from crawling the folder
There was a problem hiding this comment.
The tag will still be there, only now it's added automatically during the creation of the event.
|
@nicpenning have you got a chance to test? |
|
Not yet, but I will test as soon as I can! My initial test didn't work but I might not have had my config options or code updated properly. |
|
So far works great! Only downside is that the blob event that gets emitted to the console takes up a lot of lines (like 600K and in some cases I imagine it could be millions). It is bearable, but makes it more difficult to watch the scans progression using the default output. Going to test the strict options to ensure I only get blobs for the appropriate scope. |
|
So does FILESYSTEM == filedownload? I get all of the screenshots as PNGs in the output.ndjson but never the filedownload but shouldn't I need to set that configuration for the filedownload to use the blob feature? It doesn't seem the pdf, docx, etc.. that are in the filedownload directory are in the event at all (not just missing the blob). Running this command TL:DR - Screenshots seem to work, but the FILESYSTEM / filedownload module does not emit an event (so blob definitely does not exist). |
|
Here is a real world example: bbot -t "https://eicar.org/download-anti-malware-testfile/" -c file_blobs=true folder_blobs=true output_modules.http.siem_friendly=true modules.filedownload.base_64_encoded_file=true -m filedownload httpx gowitness Files downloaded, but none are in the events (output.ndjson). Otherwise, everything else is working as expected. |
|
Hmm I think this branch needs merged with the latest dev, so it includes this PR:
|
TheTechromancer
changed the title
|
I'm going to go ahead and merge this. We can make tweaks in a followup PR if needed. |
|
Sounds great |
Draft code things