Benchmark Tool - Enhancements - Round 1

[jakelandis]

First round of enhancements to the Logstash Benchmark Tool

• Add in some hardware stats to the output (Platform/CPU/Memory, etc) , to help contextualize the result if shared between people.
• Add a backgroud thread to add progress dots to let you know it is working, the apache test case takes a while and some feed back helps (kinda like M$'s progress bar, it means nothing, but makes you feel better).
• Add in an option for --testcase=custom and expose a --conifg-dir option such that, assuming I can figure out how to feed data in, I can use this for any test case.
• Add an option to store a csv file of the results, so that they can be source controlled, and graphed over time.
• Add in a clean command to clean up the $HOME/.logstash-benchmarks directory.

The above is not an exhaustive list (please add as needed), and is simply intended to be a place to gather thoughts for the next time someone has the time to invest in the Benchmark Tool.

[original-brownbear]

Just adding a few notes:

• Add the ability to loop initially and only start measuring once the JIT has settled down
• Make the length of a run configurable and warn on runs that were <5 min because the result is likely meaningless
• Make worker-count + other batch related params + PQ true/false configurable when invoking the benchmark
  • Same goes for JVM settings

[suyograo]

Few from my side:

• Consolidate the various benchmarks directories. Discuss if the benchmark tool needs to be split into a separate repo.
• Expose a shell wrapper to call the Java benchmarking tool instead of users having to do java -cp
• Big +1 on the ability to benchmark custom configs using custom dataset. This will be wonderful!
• Ability to export bench runs and ship to ES directly (so we can graph in Kibana)

[original-brownbear]

@suyograo @jakelandis soo :) question: What do we want first?

From my side, the Kibana/ES integration first makes some sense to finish up on the end-to-end functionality (it's kinda trivial too)?

[suyograo]

@original-brownbear +1 to ES exporter. I would tackle the custom configs next, after that.