Agent add support for multiples outputs

[ph]

This PR takes a single configuration with multiple outputs configured and will create unique configuration per output.

Multiple output example

outputs:
  default:
    type: es
    url: xxxxx
    username: xxx
    pass:  xxx
    index_name: general
  infosec1:
    type: es
    url: xxxxx
    username: xxx
    pass:  xxx
    spool:
        file: "${path.data}/spool.dat"

streams:
  - type: log
    path: /xxxx
    processors:
      - dissect:
          tokenizer: "---"
    output:
      # use_output: default
      override:
        index_name: my_service_logs
        ingest_pipeline: process_logs

  - type: log
    path: /xxxx
    output:
      # use_output: default
      index_name: mysql_access_logs

  - type: metrics/system
    username: xxxx
    pass: yyy
    output:
      use_output: infosec1
      index_name: mysql_metrics

fixes: #14445

TODO:

• update the configuration YAML for streams.

[ph]

@michalpristas Is still in draft but if you can do an early check.

[ph]

Jenkins test this please

[michalpristas]

could you add some sample transformation into description?

[ph]

@michalpristas I believe I have addresses all your concerns in the review and I also rebased the PR with the fleet branch.

[michalpristas]

👍 LGTM

[ph]

I am going to update the configuration in another PR.

[ppf2]

Can you clarify if this will allow users to configure (within a single beat) to send monitoring metrics to a remote ES monitoring cluster (output.elasticsearch) and non-monitoring metrics (actual event data) to a different output (e.g., output.kafka). Or is this use case not covered by this PR and is a different enhancement? Thx!

[ph]

@ppf2 This doesn't change the behavior of beats, "single beat" will only support a single "output". This change is only relevant to how the Agent abstract the multiples outputs cases.