[Ingest Manager] Moved from stream to dataset

[michalpristas]

What does this PR do?

Changes injected processor to events so it enriches them with dataset.* instead of stream.*

Why is it important?

elastic/package-registry#491

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

As draft until we are ready and in sync to merge

Fixes: elastic/package-registry#491
Fixes: #18826

cc @ruflin

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #18967 updated]

• Start Time: 2020-06-09T10:47:01.659+0000

• Duration: 32 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	508
Skipped	127
Total	635

Steps errors

Expand to view the steps failures

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 1 min 27 sec

  • Start Time: 2020-06-09T11:13:01.316+0000

  • log

[ph]

There a small thing missing here: dataset.dataset need to be changed for dataset.name, this will create these fields:

1. dataset.type
2. dataset.name
3. dataset.namespace.

@michalpristas @ruflin The field in the agent configuration will also need to changes? Meaning we will need to adjust the index name generation.

[michalpristas]

in configuration i would expect this

datasources:
  - namespace: testing
    use_output: default
    inputs:
    - type: docker/metrics
      streams:
        - metricset: status
          dataset: docker.status

to become this

datasources:
  - namespace: testing
    use_output: default
    inputs:
    - type: docker/metrics
      datasets:
        - metricset: status
          name: docker.status

is that correct?
but this feels weird as you can have

datasources:
  - namespace: testing
    use_output: default
    inputs:
    - type: docker/metrics
      datasets:
        - metricset: status
          name: docker.status
        - metricset: cpu
          name: docker.status

i dont recall if we agreed to handle configuration change as a separate pack of PRs or the same with this one @ruflin

[ruflin]

@michalpristas Please handle config changes as a separate PR as it also has effects in Kibana. I plan to open an issue for this. These config changes we should directly in combination with the potential removal of datasources.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

I tested this and it does not work as expected. The result is:

          "dataset" : {
            "type" : "metrics",
            "dataset" : "system.cpu",
            "namespace" : "default"
          },

But it should be

          "dataset" : {
            "type" : "metrics",
            "name" : "system.cpu",
            "namespace" : "default"
          },

I think the reason is on line 14 where it still states dataset instead of name.

[michalpristas]

oh right

[ruflin]

@michalpristas I pushed a fix with the requested change above. Lets see what CI thinks.

[ruflin]

@michalpristas Seems like this does not apply yet to agent monitoring data. Where needs this change to happen?

[ruflin]

I filed #19070 as a follow up as it is really hard to validate these changes without manual testing.

[ruflin]

Tested locally with elastic/kibana#68322 and works as expected.