Skip to content

Update to ECS 1.4.0#14844

Merged
adriansr merged 14 commits intoelastic:masterfrom
adriansr:update_ecs_1.3
Jan 10, 2020
Merged

Update to ECS 1.4.0#14844
adriansr merged 14 commits intoelastic:masterfrom
adriansr:update_ecs_1.3

Conversation

@adriansr
Copy link
Copy Markdown
Contributor

@adriansr adriansr commented Nov 28, 2019 •
edited
Loading

Updated vendored elastic/ecs to v1.3.0 v1.3.1 v1.4.0 and fields.ecs.yml from same version.

@adriansr adriansr added the ecs label Nov 28, 2019
@adriansr adriansr requested a review from a team as a code owner November 28, 2019 14:56
@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Nov 28, 2019

jenkins, test this

adriansr
adriansr commented Dec 2, 2019
View reviewed changes
Comment thread libbeat/tests/system/template/template.go Outdated
Copy link
Copy Markdown
Contributor Author

@adriansr adriansr Dec 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this change. It was complaining that there were 1006 fields after the ECS update.

Will exceed 1000 have an impact or as long as it's below 1024 we're safe?

Copy link
Copy Markdown

@urso urso Dec 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We configure the total_fields.limit per index to 10000. The default_fields limit is impacted by the indices.query.bool.max_clause_count setting (default 1024, but not modifyable via templates).

Copy link
Copy Markdown
Contributor

@ruflin ruflin Dec 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh I think touched this in the past?

Copy link
Copy Markdown
Member

@andrewkroh andrewkroh Dec 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One option to deal with this would be to add default_field: false to the new fields being added in ECS. This problem will keep occurring each time new fields are added and this could be the solution until we move to a better indexing strategy. Bumping the limit up won't work for much longer.

@adriansr adriansr requested review from a team, andrewkroh and ruflin and removed request for a team December 2, 2019 11:47
@adriansr adriansr added the review label Dec 2, 2019
ruflin
ruflin reviewed Dec 2, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I'm good with this change. We should keep in mind that this keeps increasing the size of all templates even thought he fields are not used in winlogbeat for example.

I didn't check CI but I think we should ;-)

@adriansr adriansr requested a review from ruflin December 4, 2019 10:59
@adriansr adriansr changed the title Update to ECS 1.3.0 Update to ECS 1.3.1 Dec 4, 2019
@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Dec 4, 2019

Updated to ECS 1.3.1

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Dec 9, 2019

@adriansr What are your thoughts around the field limit (see comment from @andrewkroh ).

@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Dec 10, 2019 •
edited
Loading

I misunderstood the default_fields suggestion.

I can add them to the new fields this PR adds, but this implies a change in ECS (to add default_fields in there) and a new release (say 1.3.2/1.4.0). Alternatively we can merge as is and apply the change in the next release.

I don't mind doing either.

Edit: Created a PR to elastic/ecs with the change: elastic/ecs#687

@adriansr adriansr mentioned this pull request Dec 10, 2019
Merged
@adriansr adriansr added in progress Pull request is currently in progress. and removed review labels Dec 13, 2019
@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Dec 13, 2019

This is still a work in progress as it currently depends on adding the default_field: false option to ECS (elastic/ecs#687).

A new ECS release (1.4.0?) will be needed after that.

@webmat
Copy link
Copy Markdown
Contributor

webmat commented Dec 24, 2019

elastic/ecs#687 has been merged and backported to branch 1.3.

@adriansr adriansr changed the title Update to ECS 1.3.1 Update to ECS 1.4.0 Jan 7, 2020
webmat
webmat reviewed Jan 8, 2020
View reviewed changes
Comment thread libbeat/_meta/fields.ecs.yml Outdated
Copy link
Copy Markdown
Contributor

@webmat webmat Jan 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to grab the version from elastic/ecs#718 ;-)

@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Jan 8, 2020

Jenkins, test this

@adriansr
Copy link
Copy Markdown
Contributor Author

adriansr commented Jan 10, 2020

Jenkins, test this

@adriansr adriansr merged commit cab56a3 into elastic:master Jan 10, 2020
@adriansr adriansr mentioned this pull request Jan 10, 2020
Merged
adriansr added a commit to adriansr/beats that referenced this pull request Jan 13, 2020
@adriansr
Update to ECS 1.4.0 (elastic#14844)
fef2dd8 
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch

Includes elastic/ecs#717 and  elastic/ecs#687 not in released v1.4.0

- Remove field `process.exit_code` from Metricbeat, now in ECS.

(cherry picked from commit cab56a3)
adriansr added a commit that referenced this pull request Jan 14, 2020
@adriansr
Update to ECS 1.4.0 (#14844) (#15486)
f03367c 
- Update vendored elastic/ecs and fields.ecs.yml from ECS 1.4 branch

Includes elastic/ecs#717 and  elastic/ecs#687 not in released v1.4.0

- Remove field `process.exit_code` from Metricbeat, now in ECS.

(cherry picked from commit cab56a3)
@ycombinator ycombinator mentioned this pull request Jan 14, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

@ruflin ruflin Awaiting requested review from ruflin

2 more reviewers

@webmat webmat webmat left review comments

@urso urso urso approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ecs in progress Pull request is currently in progress. v7.6.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@adriansr @ruflin @webmat @urso @andrewkroh