Introduce aliases for 7.x fields in 6.x #9283

ruflin · 2018-11-29T11:36:14Z

This introduces for aliases for fields which will exist in 7.x and a 1-1 mapping is possible. Having this fields already around makes sure we do not use them otherwise.

elasticmachine · 2018-11-29T11:36:16Z

Pinging @elastic/infrastructure

ruflin · 2018-11-29T11:37:12Z

@graphaelli I wonder if it would be helpful to start a similar PR against 6.x in apm-server to see if all the 1-1 alias do not already have an other field. If it does, CI will complain.

jsoriano · 2018-11-29T11:56:33Z

Will these fields appear in 6.x documentation without descriptions? do we want that?

There would be an option to hide them in docs if we are only adding them to not use them?

graphaelli · 2018-11-29T12:45:37Z

@ruflin already in progress, I'll add you as reviewer

ruflin · 2018-11-29T13:01:17Z

I think I need to backport this one #9283 and probably one more for the docs to look properly.

libbeat/_meta/fields.common.yml

stacey-gammon · 2018-12-10T17:15:53Z

metricbeat/docs/fields.asciidoc

I'm a little confused by the semantics here. Is beat.hostname the alias, or the field the alias will point to? Reading the above:

- name: host.hostname type: alias path: beat.hostname

makes me think that host.hostname is the alias that is going to point to the real field beat.hostname. I know the ECS field name is host.hostname. Is the plan in 6.x it'll be the alias, and in 7.x, it'll be the real field when the mapping support many : 1 (the 1 being the ECS field, the many being all variations that are used across all the beats).

Can you let me know the specific field that will require the many : 1 mapping? I thought I recalled Boaz saying something about filebeat, but looking at the fields (well at least the ones with host in them) I don't see where the issue specifically is. Just looks like all the beats have beat.hostname which would map to host.hostname, which should still be 1:1? Even though beat.hostname is a field in many different indices... ?

Sorry for so many questions, just trying to wrap my head around all this as I think about what should be tested!

The fields that are aliased here are for 6.x and are only 1:1 mappings to make them compatible with 7.x data for Logging UI and Infrastructure UI.

For the many:1 mappings which are only introduce in 7.x you can find some examples here:

Convert apache2.access to ECS - Take 2 #9245

Convert Filebeat nginx.access to ECS #9081

This form apache and nginx and you will see that these have many fields in common and the old fields are mapped to the same new fields.

@stacey-gammon Here's my attempt at clarifying as well.

For 6.x and for 7.x, the aliases are flipping direction indeed.

6.x: new fields introduced as aliases to the old field they will eventually replace. This is for forward compatibility. We will start being able to depend on the new field names in the later releases in the 6.x line for those. This is only applicable for 1:1 mappings, however (explained below).

7.x: For all fields being renamed (1:1 or many to 1), the ECS field is now the canonical field holding the data, and old fields alias to the ECS field.

Many to 1 mappings: Filebeat modules are indeed where this is happening the most, but there may be other places. All modules used to nest their fields under a namespace [module name].[dataset].[field]. In many cases, different modules had perfectly equivalent fields: apache2.access.url and nginx.access.url. These two examples are becoming url.original. In 6.x we can't create a forward compatible alias to all of these values, as the alias can only point to one field. However in 7.x we'll have all of these old fields be aliases to the new canonical field. Nothing prevents having many alias fields pointing to the same field.

graphaelli

Planning to introduce an ecs-migration.yml log too?

libbeat/_meta/fields.common.yml

dev-tools/ecs-migration.yml

webmat

This looks ok overall. Here's a few points, though:

ECS doesn't have cloud.project.id
Please review my question about beat.name & agent.type. I don't think this is the right mapping.

Not a blocker for this PR, but we must address for 6.6:

You mentioned in a comment that you'll need to backport the PRs to make the asciidoc for aliases to look good, but you pasted this PR's url instead of the correct PRs :-) The correct links are PR #9269 and issue #9288. I've listed them both under the 6.x section in #8655.

webmat · 2018-12-11T15:18:20Z

dev-tools/ecs-migration.yml

I didn't realize we had both fileset.module and metricset.module becoming event.module. But I see you've treated them as two 1:1 mappings, in different indices. Good stuff.

libbeat/_meta/fields.common.yml

webmat · 2018-12-11T16:31:17Z

Gil makes good points with some of the missing aliases:

beat.timezone ~ event.timezone
docker.container.id ~ container.id

Looking at the list of libbeat common fields, I've spotted a few more:

beat.version ~ agent.version
docker.container.labels ~ container.labels

One we can't tackle because it's an object is fields ~ labels, unfortunately.

graphaelli · 2018-12-11T16:37:10Z

@webmat container.labels has the same problem as labels right? in apm-server, we planned to duplicate those in 6.x - that is write them to both places.

agent.version means something different in APM, if you're going to add this I request that it be done per beat.

webmat · 2018-12-11T19:28:53Z

@graphaelli Ah you're right about container.labels. I had a lapse and was thinking about the semantics of tags (array of keywords).

ruflin · 2018-12-11T19:29:52Z

cloud.project.id is not in ECS and that is ok
I added container.id.
I will not add container.labels as they are not used by Infrastructure & Logging UI. I will leave it to apm-server to handle this.
beat.version was left out on purpose as we don't use it anywhere. And now that I see it could cause issues, event better.
beat.timezone -> event.timezone added
I removed beat.name mapping as I agree it's the wrong one here and we don't need to migrate it.

webmat

LGTM, once the build is green 👍

webmat · 2018-12-11T20:53:28Z

dev-tools/ecs-migration.yml

Flip to alias6: false?

Good find. Remove it and added event.timezone and container.id instead as it was missing.

webmat · 2018-12-11T20:56:08Z

Still looking good after the rebase, but I noticed the alias6 on beat.name. Otherwise all good

webmat · 2018-12-11T21:28:08Z

dev-tools/ecs-migration.yml

You're right. This whole section needed to go, not just the alias6 :-)

ruflin · 2018-12-12T10:23:41Z

jenkins, test this

ruflin · 2018-12-12T10:23:55Z

Seems like the LS integration tests were flaky which I haven't seen before. Retrying.

ruflin · 2018-12-12T14:40:36Z

jenkins, test this

This introduces for aliases for fields which will exist in 7.x and a 1-1 mapping is possible. Having this fields already around makes sure we do not use them otherwise. need at least 6.4 for testing trying out if snapshot trick works reset docker compose update more fields.go files rebase to get 6.4 environment remove not needed fields add migraton field update fields.yml again update fields rebase update migration document add alias docs

webmat

Rebase still looks good

ruflin · 2018-12-12T16:16:04Z

The failures are very similar to before so I'm starting to guess it's related. Unfortunately so far I couldn't reproduce it locally.

ruflin · 2018-12-12T16:16:11Z

jenkins, test this

ruflin · 2018-12-12T19:34:13Z

Strangely now all tests except a new flaky one went green. Merged for now but will keep an eye on it.

This introduces for aliases for fields which will exist in 7.x and a 1-1 mapping is possible. Having this fields already around makes sure we do not use them otherwise.

ruflin added in progress Pull request is currently in progress. ecs Team:Integrations Label for the Integrations team labels Nov 29, 2018

ruflin mentioned this pull request Nov 29, 2018

Transition Beats to ECS #8655

Closed

alvarolobato assigned ruflin Dec 3, 2018

ruflin force-pushed the introduce-aliases branch from 29c68ca to 4cf3014 Compare December 10, 2018 08:11

ruflin commented Dec 10, 2018

View reviewed changes

libbeat/_meta/fields.common.yml Outdated Show resolved Hide resolved

ruflin added review and removed in progress Pull request is currently in progress. labels Dec 10, 2018

ruflin mentioned this pull request Dec 10, 2018

Migrate add_docker_metadata to ECS #9412

Merged

stacey-gammon reviewed Dec 10, 2018

View reviewed changes

graphaelli reviewed Dec 11, 2018

View reviewed changes

libbeat/_meta/fields.common.yml Outdated Show resolved Hide resolved

webmat reviewed Dec 11, 2018

View reviewed changes

dev-tools/ecs-migration.yml Outdated Show resolved Hide resolved

webmat suggested changes Dec 11, 2018

View reviewed changes

graphaelli reviewed Dec 11, 2018

View reviewed changes

libbeat/_meta/fields.common.yml Outdated Show resolved Hide resolved

libbeat/_meta/fields.common.yml Outdated Show resolved Hide resolved

webmat approved these changes Dec 11, 2018

View reviewed changes

ruflin force-pushed the introduce-aliases branch from 230f221 to fadb65f Compare December 11, 2018 20:48

webmat reviewed Dec 11, 2018

View reviewed changes

dev-tools/ecs-migration.yml Outdated

Copy link

Contributor

webmat Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. This whole section needed to go, not just the alias6 :-)

webmat approved these changes Dec 11, 2018

View reviewed changes

ruflin force-pushed the introduce-aliases branch from c07a437 to f5adede Compare December 11, 2018 21:40

ruflin added the v6.6.0 label Dec 12, 2018

ruflin force-pushed the introduce-aliases branch 2 times, most recently from e61220a to dffc8c0 Compare December 12, 2018 13:38

ruflin added 3 commits December 12, 2018 16:03

update fields

24cd94d

rebase again

8652806

ruflin force-pushed the introduce-aliases branch from dffc8c0 to 8652806 Compare December 12, 2018 15:05

webmat mentioned this pull request Dec 12, 2018

Rename beat.timezone to event.timezone in add_locale processor #9458

Merged

webmat approved these changes Dec 12, 2018

View reviewed changes

ruflin merged commit c63c98b into elastic:6.x Dec 12, 2018

ruflin deleted the introduce-aliases branch December 12, 2018 19:33

This was referenced Dec 17, 2018

Infrastructure UI ECS Migration elastic/kibana#26618

Closed

Logging UI ECS Migration elastic/kibana#27284

Closed

Introduce aliases for 7.x fields in 6.x #9283

Introduce aliases for 7.x fields in 6.x #9283

Uh oh!

Conversation

ruflin commented Nov 29, 2018

Uh oh!

elasticmachine commented Nov 29, 2018

Uh oh!

ruflin commented Nov 29, 2018

Uh oh!

jsoriano commented Nov 29, 2018

Uh oh!

graphaelli commented Nov 29, 2018

Uh oh!

ruflin commented Nov 29, 2018

Uh oh!

Uh oh!

stacey-gammon Dec 10, 2018

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

webmat Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

graphaelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

webmat left a comment

Choose a reason for hiding this comment

Uh oh!

webmat Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

webmat commented Dec 11, 2018

Uh oh!

graphaelli commented Dec 11, 2018

Uh oh!

webmat commented Dec 11, 2018

Uh oh!

ruflin commented Dec 11, 2018

Uh oh!

webmat left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webmat Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

webmat commented Dec 11, 2018

Uh oh!

webmat Dec 11, 2018

Choose a reason for hiding this comment

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

webmat left a comment

Choose a reason for hiding this comment

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

ruflin commented Dec 12, 2018

Uh oh!

Reviewers

Assignees

webmat left a comment •

edited

Loading