Remove duplicated fields

[jsoriano]

What does this PR do?

Remove duplicated fields.

This is important to avoid unexpected results, when definitions are different any of them could be applied.

This is also a requirement to use package-spec 1.8.0.

See elastic/package-spec#309.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Related to Validate that fields are defined only once per data stream package-spec#309.

Screenshots

[elasticmachine]

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-05-12T07:54:59.554+0000

• Duration: 94 min 41 sec

Test stats 🧪

Test	Results
Failed	6
Passed	4721
Skipped	7
Total	4734

Test errors

Expand to view the tests failures

Check integrations / auditd / auditd: check / system test: default – auditd.log

Expand to view the error details

 null 

Expand to view the stacktrace

 one or more errors found in documents stored in logs-auditd.log-ep data stream: [0] field "container.name" is undefined
[1] field "container.runtime" is undefined 

Check integrations / system / system: check / system test: default – system.process

Expand to view the error details

 null 

Expand to view the stacktrace

 one or more errors found in documents stored in metrics-system.process-ep data stream: [0] field "process.args" is undefined
[1] field "process.command_line" is undefined
[2] field "process.cpu.pct" is undefined
[3] field "process.cpu.start_time" is undefined
[4] field "process.executable" is undefined
[5] field "process.memory.pct" is undefined
[6] field "process.name" is undefined
[7] field "process.parent.pid" is undefined
[8] field "process.pgid" is undefined
[9] field "process.pid" is undefined
[10] field "process.state" is undefined
[11] field "process.working_directory" is undefined
[12] field "user.name" is undefined 

Check integrations / awsfargate / awsfargate: check / static test: Verify sample_event.json – awsfargate.task_stats

Expand to view the error details

 null 

Expand to view the stacktrace

 one or more errors found in document: [0] field "container.id" is undefined
[1] field "container.image.name" is undefined
[2] field "container.labels.com_amazonaws_ecs_cluster" is undefined
[3] field "container.labels.com_amazonaws_ecs_container-name" is undefined
[4] field "container.labels.com_amazonaws_ecs_task-arn" is undefined
[5] field "container.labels.com_amazonaws_ecs_task-definition-family" is undefined
[6] field "container.labels.com_amazonaws_ecs_task-definition-version" is undefined
[7] field "container.name" is undefined 

Check integrations / rabbitmq / rabbitmq: check / static test: Verify sample_event.json – rabbitmq.exchange

Expand to view the error details

 null 

Expand to view the stacktrace

 one or more errors found in document: [0] field "user.name" is undefined 

Check integrations / haproxy / haproxy: check / static test: Verify sample_event.json – haproxy.info

Expand to view the error details

 null 

Expand to view the stacktrace

 one or more errors found in document: [0] field "process.pid" is undefined 

Check integrations / haproxy / haproxy: check / static test: Verify sample_event.json – haproxy.stat

Expand to view the error details

 null 

Expand to view the stacktrace

 one or more errors found in document: [0] field "process.pid" is undefined 

Steps errors

Expand to view the steps failures

Test integration: auditd

• Took 1 min 10 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

Test integration: awsfargate

• Took 1 min 33 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

Test integration: haproxy

• Took 2 min 58 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

Check integration: juniper

• Took 0 min 2 sec . View more details here
• Description: ../../build/elastic-package check -v

Boot up the Elastic stack

• Took 2 min 5 sec . View more details here
• Description: ../../build/elastic-package stack up -d -v --version 7.14.0

Check integration: kubernetes

• Took 0 min 5 sec . View more details here
• Description: ../../build/elastic-package check -v

Test integration: rabbitmq

• Took 1 min 35 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

Test integration: system

• Took 2 min 6 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (237/237)	💚
Files	96.314% (392/407)	👎 -0.062
Classes	96.314% (392/407)	👎 -0.062
Methods	88.111% (4017/4559)	👎 -0.09
Lines	89.283% (69432/77766)	👍 0.342
Conditionals	100.0% (0/0)	💚

[andrewkroh]

I wouldn't expect any adds or removes in the READMEs, only modifications to descriptions or types due to the duplicate removal.

[jsoriano]

Adds are going to happen, because of elastic/elastic-package#814, we were installing fields that were not documented.

But yes, removals shouldn't happen. There are still some other things I have to polish here.

[jsoriano]

@andrewkroh at the end having adds and removes was a a good indicator that something was wrong 🙂

elastic/package-spec#331
#3392

[jsoriano]

We are delaying removal of duplicated fields till we clarify some issues (elastic/package-spec#331) closing this by now.