Feature: Better Logging Architecture

[KillerInstinct]

There are multiple enhancements that could be done to the current logging architecture.

• Rolling logs
  After about 1000 analysis (or 20 with debug information) the cuckoo.log begins to get a bit overwhelming to look at, some error logs do not specify which task failed which can make grepping for logs hard. Especially when concurrent tasks are running. This goes into the next point...
• Ensure Task ID's are printed in as many error/warning logs as possible. Several locations in plugins.py could benefit from logging the task ID for processing/signature errors:
  https://github.com/cuckoobox/cuckoo/blob/master/lib/cuckoo/core/plugins.py#L195
  https://github.com/cuckoobox/cuckoo/blob/master/lib/cuckoo/core/plugins.py#L358
  https://github.com/cuckoobox/cuckoo/blob/master/lib/cuckoo/core/plugins.py#L560
  Another example (and there are others really): https://github.com/cuckoobox/cuckoo/blob/master/modules/processing/behavior.py#L249
• Add a log handler to process.py. Currently there is only STDOUT. If you want to reprocess an analysis, there are no retainable logs unless you pipe it to a file and after reprocessing, the logs for that task in cuckoo.log may not be applicable anymore. For people using process.py in auto mode, there is no logging at all unless you force it with something like an upstart job, or redirecting stdout to a file, etc. Would be wonderful to have this be logged to its own file.
• Perhaps add a configuration file for logging. I'd love to see the ability to log to syslog for something like Splunk or ElasticSearch.
• Task aware logging. All logs specific to a particular task should be monitored in a task-by-task basis so that we can dump them into MongoDB. We could then download/display them in the Admin tab (or similar). This would mostly eliminate the need to log into the server to get logs, and would make Cuckoo be more 'appliance' like.

[jbremer]

Agreed on most points. @rep already proposed the task aware logging a while ago but we haven't put it to production yet.
In addition to the points listed here I'd also like to see a per-task error.log which contains a dump of all exceptions that happened for this particular task. Naturally each task should have no exceptions at all, but it has happened various times where incorrect usage of 3rd party libraries caused exceptions, e.g., dpkt, pefile, volatility, etc.

[jbremer]

Btw, in the case of rolling logs, we do have to make sure that logs are not magically deleted. E.g., I still may want to see the logs of an analysis half a million analyses later ;)

[KillerInstinct]

Stumbled back across then when I was checking Issues I was involved in. I ended up coding the third point because an upstart job fails to account for when you want to reprocess and analysis, you're still only left with logging to STDOUT. This solves that.

Relevant commits:
spender-sandbox/cuckoo-modified@e22b9ad
spender-sandbox/cuckoo-modified@00f176c

Feel free to use/improve.

[jbremer]

Having forgotten about this issue, some of the suggestions have been implemented by now.
In particular: more task ID logging, per-task logging, process.py logging (in #863), and the task aware logging.
I guess the rest have yet to be done :-)