[Synthetics] - add meta information to monitoring data

[jalogisch]

Describe the feature:

When you create monitoring for different URLs (as example) in Synthetics, you want to have the ability to add additional fields and information to the data that is stored in elasticsearch as a result of these monitors.

Describe a specific use case for the feature:

When you monitor different applications or business areas, you want to have the ability to have a single field that allows you across all your data to get the information for this specific business area.

So you want to add to all relevant data to your website one field with the same value. - area:website - that allows you to search all available data, web server logs, database, uptime checks, and application logs for everything related to that specific area.
In addition, you could allow access based on this field to specific teams and contractors only.

All of this would allow having the important data in a single place, creating dashboards with sourcing different logs and data sources to create one big picture that enables operators to get a quick overview.

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[paulb-elastic]

Our current approach for something similar is to use tags to group similar monitors together (multiple tags can be defined per monitor). How does this requirement differ from that?

[philippkahr]

@paulb-elastic ah I see what you want todo. That is not the plan here. This is a bit of an inconsistency across all integrations, nonetheless most of them supply a processor configuration. Therefore you can run a certain amount of processors (usually those: https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html and do things, like add custom fields.

[paulb-elastic]

Thanks for clarifying.

As you’re aware, we don’t expose any of the Synthetics Integration configuration, and this is by design. We’ve been careful to ensure configurations are managed from within the Synthetics UI (earlier, Uptime Monitor Management), so that we can abstract the monitor configuration from the underlying implementation and be sure things are set up correctly. That’s effectively why we have an opinionated UI (or definition for Project Monitors) in Synthetics and force all configurations through there.

As such, adding the above capability is more likely to end up being a monitor configuration, either through the UI, or as a Project Monitor configuration for example (via the DSL).

I’ll move this over to the Synthetics repo and put it on our backlog, but for transparency, we’re focusing on the core requirements for Synthetics right now, so won’t be able to look at this any time soon.

[vigneshshanmugam]

Related #750

[dejongm]

We're migrating to Synthetics from Heartbeat and this feature is very much a requirement for us to move forward without implementing some messy workarounds. Can we get a status, regarding progress or timeline on this?

[shahzad31]

@dejongm @jalogisch do you expect those meta fields to be insert at the root of the documents? or would it suffice if they are nested let's say under a field meta : {area: website} it might be bit difficult for us to append at root , since that may have mappings repercussions. Appending under meta or lets say labels would be pretty easy and can be done in near term.

[philippkahr]

I would rather do labels as this is in ECS and I have seen it more commonly in the wild with like labels.team: abc.

[jalogisch]

I second the ECS Compliant solution.

[herrBez]

If you need to rename the labels to the expected fields, you can always use the Ingest Pipeline synthetics@custom.

[dejongm]

I agree to adhere with the ECS and nest them under labels.