Set pipeline.buffer.type to heap by default

[robbavey]

In #16054, we introduced a new setting to allow users to set the allocation of memory for certain input plugins to use either direct or heap memory.

We should consider making this the default setting for Logstash 9.0

Tasks

Preview Give feedback

1. on 8.x, when pipeline.buffer.type is not set, we should emit a deprecation log guiding to do one of two things before upgrading to 9.x: either (1) set pipeline.buffer.type: heap and tune heap size upward in advance of upgrading to 9.x or (2) set pipeline.buffer.type: direct to lock in the existing behaviour. Log deprecation warn if memory buffer type not defined #16498
2. on main, set default for pipeline.buffer.type to heap Default buffer type to 'heap' for 9.0 #16500

[flexitrev]

Would this be a breaking change if we are only changing the default?

[jsvd]

Any deployment out there consuming from Agent or Beats with a heap usage above 75% (or even 50%) can potentially crash on upgrade as any direct memory allocation from beats/agent/http/tcp inputs would now be done on the heap.

[robbavey]

Re-opened, as the deprecation warning sub-issue has not been closed