Skip to content

Remove duplicated fields #3316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jsoriano wants to merge 33 commits into from
Closed

Remove duplicated fields #3316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
e22eec9
Activemq
jsoriano May 9, 2022
8f242da
Akamai
jsoriano May 9, 2022
05e4ca7
Apache
jsoriano May 9, 2022
c2c2f96
Auditd
jsoriano May 9, 2022
6eed806
Auth0
jsoriano May 9, 2022
64d2aa9
AWS
jsoriano May 9, 2022
54797ac
Merge remote-tracking branch 'origin/main' into remove-duplicated-fields
jsoriano May 10, 2022
2de2406
Recover missing docs
jsoriano May 10, 2022
f0d81fc
Remove duplicates in ecs files
jsoriano May 10, 2022
6a2ab61
Remove fields and sub-fields defined in ECS
jsoriano May 10, 2022
fccd3de
More duplicates
jsoriano May 10, 2022
e4bd8c6
More duplicates in ECS files
jsoriano May 10, 2022
43361a4
Remove empty files
jsoriano May 10, 2022
e98bde5
Remove duplicated from agent.yml
jsoriano May 10, 2022
c270995
More fields
jsoriano May 10, 2022
51686b3
Merge remote-tracking branch 'origin/main' into remove-duplicated-fields
jsoriano May 10, 2022
fa356ff
Merge remote-tracking branch 'origin/main' into remove-duplicated-fields
jsoriano May 11, 2022
166a710
Cassandra
jsoriano May 11, 2022
89b4298
More duplicated fields
jsoriano May 11, 2022
d254c06
Build readmes
jsoriano May 11, 2022
107d7a1
Merge remote-tracking branch 'origin/main' into remove-duplicated-fields
jsoriano May 11, 2022
cc1aa5d
Revert some removals
jsoriano May 11, 2022
2b2f05d
Use elastic-package from master for testing
jsoriano May 11, 2022
876186e
Adjustments in AWS
jsoriano May 11, 2022
96a9841
Recover host.containerized
jsoriano May 11, 2022
b4c3119
Fixes for azure
jsoriano May 11, 2022
00dc316
Cisco
jsoriano May 11, 2022
6482a5a
Revert more deletions
jsoriano May 11, 2022
badb360
Fix netflow
jsoriano May 11, 2022
da3a906
Revert changes in network_traffic
jsoriano May 11, 2022
25b9989
Revert removal of dns.answers
jsoriano May 11, 2022
84a335a
Fix format
jsoriano May 11, 2022
71c9f55
Merge remote-tracking branch 'origin/main' into remove-duplicated-fields
jsoriano May 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 3 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ require (
github.com/elastic/go-sysinfo v1.7.1 // indirect
github.com/elastic/go-ucfg v0.8.4 // indirect
github.com/elastic/go-windows v1.0.1 // indirect
github.com/elastic/package-spec v1.7.1 // indirect
github.com/elastic/package-spec v1.8.0 // indirect
github.com/emirpasic/gods v1.12.0 // indirect
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
Expand Down Expand Up @@ -164,3 +164,5 @@ require (
sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)

replace github.com/elastic/elastic-package => github.com/elastic/elastic-package v0.48.1-0.20220511104207-12db194afa2d
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,8 +409,8 @@ github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5/go.mod h1:qssHWj6
github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/elastic/elastic-package v0.48.2 h1:sIF8Shfsu+J96bllzEMgc7BDhpaE2b4gCXziYwyNRXw=
github.com/elastic/elastic-package v0.48.2/go.mod h1:tUXKdvUNGgV9myTfbeGmI4LNMUjuILAlXF8NfZNiCHw=
github.com/elastic/elastic-package v0.48.1-0.20220511104207-12db194afa2d h1:p38gJ7OEa58gawB8XDQ4sdzFfsHQlxvlmXh/vaiOEZ8=
github.com/elastic/elastic-package v0.48.1-0.20220511104207-12db194afa2d/go.mod h1:0xJUNGxEvB3+scNrdVf1TMXUMjpL6VGyLYQAIfMDZss=
github.com/elastic/go-elasticsearch/v7 v7.17.1 h1:49mHcHx7lpCL8cW1aioEwSEVKQF3s+Igi4Ye/QTWwmk=
github.com/elastic/go-elasticsearch/v7 v7.17.1/go.mod h1:OJ4wdbtDNk5g503kvlHLyErCgQwwzmDtaFC4XyOxXA4=
github.com/elastic/go-licenser v0.3.1/go.mod h1:D8eNQk70FOCVBl3smCGQt/lv7meBeQno2eI1S5apiHQ=
Expand All @@ -427,8 +427,8 @@ github.com/elastic/go-windows v1.0.1 h1:AlYZOldA+UJ0/2nBuqWdo90GFCgG9xuyw9SYzGUt
github.com/elastic/go-windows v1.0.1/go.mod h1:FoVvqWSun28vaDQPbj2Elfc0JahhPB7WQEGa3c814Ss=
github.com/elastic/package-registry v1.8.0 h1:c2nUbBZct3c2LZ6uw0HotB+11PmYM8xh0ynvyeuzFBY=
github.com/elastic/package-registry v1.8.0/go.mod h1:zh8h1v9v2VYBQvlZK2KoD/uDJlYC7re5PLf4eDALEFA=
github.com/elastic/package-spec v1.7.1 h1:Q2THMEnG4sRy+XSty16S2JLnVsROq4Ddo80WgQJzbo0=
github.com/elastic/package-spec v1.7.1/go.mod h1:KzGTSDqCkdhmL1IFpOH2ZQNSSE9JEhNtndxU3ZrQilA=
github.com/elastic/package-spec v1.8.0 h1:/5P4SwQhJgfULRg1b7I83TOzij4/L+J39o1LJiJTiJ0=
github.com/elastic/package-spec v1.8.0/go.mod h1:KzGTSDqCkdhmL1IFpOH2ZQNSSE9JEhNtndxU3ZrQilA=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 h1:yUdfgN0XgIJw7foRItutHYUIhlcKzcSf5vDpdhQAKTc=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
Expand Down
8 changes: 0 additions & 8 deletions packages/activemq/data_stream/audit/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,14 @@
name: user.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.os.full
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.version
- external: ecs
Expand Down
8 changes: 0 additions & 8 deletions packages/activemq/data_stream/broker/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,14 @@
name: user.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.os.full
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.version
- external: ecs
Expand Down
8 changes: 0 additions & 8 deletions packages/activemq/data_stream/log/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,14 @@
name: user.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.os.full
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.version
- external: ecs
Expand Down
8 changes: 0 additions & 8 deletions packages/activemq/data_stream/queue/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,14 @@
name: user.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.os.full
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.version
- external: ecs
Expand Down
8 changes: 0 additions & 8 deletions packages/activemq/data_stream/topic/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,14 @@
name: user.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.os.full
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.version
- external: ecs
Expand Down
2 changes: 0 additions & 2 deletions packages/akamai/data_stream/siem/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@
external: ecs
- name: client.geo.continent_name
external: ecs
- name: client.geo.country_iso_code
external: ecs
- name: client.geo.region_iso_code
external: ecs
- name: client.geo.location
Expand Down
8 changes: 0 additions & 8 deletions packages/apache/data_stream/access/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,14 @@
name: user.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.device.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.name
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.original
- external: ecs
name: user_agent.os.full
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.name
- external: ecs
name: user_agent.os.version
- external: ecs
Expand Down
5 changes: 0 additions & 5 deletions packages/apache/data_stream/error/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,6 @@
- name: '@timestamp'
type: date
description: Event timestamp.
- name: tags
description: List of keywords used to tag each event.
example: '["production", "env2"]'
ignore_above: 1024
type: keyword
- name: event.module
type: constant_keyword
description: Event module
Expand Down
34 changes: 0 additions & 34 deletions packages/auditd/data_stream/log/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,34 +54,6 @@
- name: image.id
type: keyword
description: Image ID for the cloud instance.
- name: container
title: Container
group: 2
description: 'Container fields are used for meta information about the specific container that is the source of information.

These fields help correlate data based containers from any runtime.'
type: group
fields:
- name: id
level: core
type: keyword
ignore_above: 1024
description: Unique container id.
- name: image.name
level: extended
type: keyword
ignore_above: 1024
description: Name of the image the container was built on.
- name: labels
level: extended
type: object
object_type: keyword
description: Image labels.
- name: name
level: extended
type: keyword
ignore_above: 1024
description: Container name.
- name: host
title: Host
group: 2
Expand All @@ -90,12 +62,6 @@
ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
type: group
fields:
- name: architecture
level: core
type: keyword
ignore_above: 1024
description: Operating system architecture.
example: x86_64
- name: domain
level: extended
type: keyword
Expand Down
4 changes: 1 addition & 3 deletions packages/auditd/data_stream/log/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
- external: ecs
name: container.name
- external: ecs
name: container.runtime
name: container
- external: ecs
name: destination.address
- external: ecs
Expand Down
3 changes: 0 additions & 3 deletions packages/auditd/data_stream/log/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,6 @@
type: keyword
description: |
The first argument to the system call.
- name: a0
description: The first argument to the system call.
type: keyword
- name: addr
type: ip
- name: rport
Expand Down
19 changes: 0 additions & 19 deletions packages/auditd/data_stream/log/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,25 +24,6 @@
type: keyword
description: |
Name of the group.
- name: effective
type: group
fields:
- name: id
type: keyword
description: |
One or multiple unique identifiers of the user.
- name: name
type: keyword
description: |
Short name or login of the user.
- name: group.id
type: keyword
description: |
Unique identifier for the group on the system/platform.
- name: group.name
type: keyword
description: |
Name of the group.
- name: filesystem
type: group
fields:
Expand Down
7 changes: 7 additions & 0 deletions packages/auditd/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,10 +183,17 @@ An example event for `log` looks as following:
| cloud.project.id | Name of the project in Google Cloud. | keyword |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
| cloud.region | Region in which this host is running. | keyword |
| container.cpu.usage | Percent CPU used which is normalized by the number of CPU cores and it ranges from 0 to 1. Scaling factor: 1000. | scaled_float |
| container.disk.read.bytes | The total number of bytes (gauge) read successfully (aggregated from all disks) since the last metric collection. | long |
| container.disk.write.bytes | The total number of bytes (gauge) written successfully (aggregated from all disks) since the last metric collection. | long |
| container.id | Unique container id. | keyword |
Copy link
Member

@andrewkroh andrewkroh May 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't expect any adds or removes in the READMEs, only modifications to descriptions or types due to the duplicate removal.

Copy link
Member Author

@jsoriano jsoriano May 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds are going to happen, because of elastic/elastic-package#814, we were installing fields that were not documented.

But yes, removals shouldn't happen. There are still some other things I have to polish here.

Copy link
Member Author

@jsoriano jsoriano May 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh at the end having adds and removes was a a good indicator that something was wrong 🙂

elastic/package-spec#331
#3392

| container.image.name | Name of the image the container was built on. | keyword |
| container.image.tag | Container image tags. | keyword |
| container.labels | Image labels. | object |
| container.memory.usage | Memory usage percentage and it ranges from 0 to 1. Scaling factor: 1000. | scaled_float |
| container.name | Container name. | keyword |
| container.network.egress.bytes | The number of bytes (gauge) sent out on all network interfaces by the container since the last metric collection. | long |
| container.network.ingress.bytes | The number of bytes received (gauge) on all network interfaces by the container since the last metric collection. | long |
| container.runtime | Runtime managing this container. | keyword |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down
3 changes: 0 additions & 3 deletions packages/auth0/data_stream/logs/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,6 @@
- name: strategy_type
type: keyword
description: Type of strategy involved in the event.
- name: log_id
type: keyword
description: Unique ID of the event.
- name: is_mobile
type: boolean
description: Whether the client was a mobile device (true) or desktop/laptop/server (false).
Expand Down
2 changes: 1 addition & 1 deletion packages/auth0/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ The Auth0 logs dataset provides events from Auth0 log stream. All Auth0 log even
| auth0.logs.data.location_info.latitude | Global latitude (horizontal) position. | keyword |
| auth0.logs.data.location_info.longitude | Global longitude (vertical) position. | keyword |
| auth0.logs.data.location_info.time_zone | Time zone name as found in the [tz database](https://www.iana.org/time-zones). | keyword |
| auth0.logs.data.log_id | Unique ID of the event. | keyword |
| auth0.logs.data.log_id | Unique log event identifier | keyword |
| auth0.logs.data.login.completedAt | Time at which the operation was completed | date |
| auth0.logs.data.login.elapsedTime | Number of milliseconds the operation took to complete. | long |
| auth0.logs.data.login.initiatedAt | Time at which the operation was initiated | date |
Expand Down
56 changes: 0 additions & 56 deletions packages/aws/data_stream/billing/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,59 +1,3 @@
- name: cloud
title: Cloud
group: 2
description: Fields related to the cloud or infrastructure the events are coming from.
footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.'
type: group
fields:
- name: account.id
level: extended
type: keyword
ignore_above: 1024
description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment.

Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.'
example: 666777888999
- name: availability_zone
level: extended
type: keyword
ignore_above: 1024
description: Availability zone in which this host is running.
example: us-east-1c
- name: instance.id
level: extended
type: keyword
ignore_above: 1024
description: Instance ID of the host machine.
example: i-1234567890abcdef0
- name: instance.name
level: extended
type: keyword
ignore_above: 1024
description: Instance name of the host machine.
- name: machine.type
level: extended
type: keyword
ignore_above: 1024
description: Machine type of the host machine.
example: t2.medium
- name: provider
level: extended
type: keyword
ignore_above: 1024
description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
example: aws
- name: region
level: extended
type: keyword
ignore_above: 1024
description: Region in which this host is running.
example: us-east-1
- name: project.id
type: keyword
description: Name of the project in Google Cloud.
- name: image.id
type: keyword
description: Image ID for the cloud instance.
- name: container
title: Container
group: 2
Expand Down
Loading