[Streams 🌊] Enrichment - Add support for date processor

[tonyghiani]

📓 Summary

Part of https://github.com/elastic/streams-program/issues/38

This work adds the date processor along with the dissect and grok ones in the enrichment section.
It scales well following the current folder structure, but we should definitely polish it a bit more once more processors get added, such as getting the right form component, improve form state derivation, etc.

demo.mov

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[flash1293]

The tag input looks nice, but it is not suited for editing date formats. I tried to use it with some actual data, and you need to submit the tag to run the simulation, but then you notice you made a mistake and have to delete the tag again to make a change. I would say either let's just go with a single format (the user can create multiple processors if necessary) or with multiple inputs like grok does it.

This is a bit confusing - it says it updates the input time field, but then it says it defaults to @timestamp. Seems like the latter is true:

With this I think it's a good starting point, but we need to add a lot of other things as well to make it really nice (kind of similar to the grok integration). Those can be follow-ups though.

[Kerry350]

Taking this over from Marco.

I would say either let's just go with a single format (the user can create multiple processors if necessary) or with multiple inputs like grok does it.

I'll change this to a single format for now 👍

This is a bit confusing - it says it updates the input time field, but then it says it defaults to @timestamp. Seems like the latter is true

I'll update the wording. The latter is true, from the docs:

By default, the date processor adds the parsed date as a new field called @timestamp.

[Kerry350]

@flash1293

I'm just comparing this to the Stack Management implementation (which is likely why we have the incorrect information about the input field being updated in place in both places).

Thinking more about formats, for all that we can apply multiple processors, I think it makes sense to have a UX where we can just do this once since they are an array of formats and applied sequentially.

Is your concern with the current implementation (matching Stack Management also) that if you make a mistake on an earlier "tag" / format, you need to remove all of them and start again?

We could also allow a comma separated list (or does that conflict with certain formats 🤔).

I'm not sure we need all the ceremony of multiple inputs (in the way the Grok processor does it).

[flash1293]

Is your concern with the current implementation (matching Stack Management also) that if you make a mistake on an earlier "tag" / format, you need to remove all of them and start again?

Exactly - try to use it to type a format from scratch manually and you will notice. It's really hard to use if you are not nailing it with the first try.

We could also allow a comma separated list (or does that conflict with certain formats 🤔).

It does conflict with certain formats, e.g. it's common for the milli seconds to be separated by comma.

I'm not sure we need all the ceremony of multiple inputs (in the way the Grok processor does it).

I would be happy if we could avoid it, but it needs to be possible to type, see the result, then fix. A single input would solve it as well for now, we can also come back to this, but the current solution isn't feasible I think.

[Kerry350]

@flash1293 Understood 👌

I've updated the field descriptions to be accurate (reflect the docs). And I've changed formats to just a singular input that takes one format (multiple processors can be used for multiple formats).

This isn't quite ready for re-review yet as there seems to be a bug whereby changes to optional fields don't retrigger the simulation. I'll ping you when it's ready 👍

[Kerry350]

@flash1293 Should be good to go now 👍

[flash1293]

All works as expected now, thanks for the updates!

Some more general thoughts - wdyt about this @LucaWintergerst and @Kerry350 ?

Using this made me realize that it's actually quite awkward how the processors are cut here - most of the time you parse out a weird date format from the message, then you want to normalize it and stuff it into @timestamp, but we force users to put it into a separate field first, use the date processor there and then put it where it actually belongs, and then removing the temporary field.

E.g. here: https://github.com/elastic/integrations/blob/b4dc45ddfceda721a029414485872d249ab17473/packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_policy_diagnostics.yml#L12-L53

Maybe we could fuse these common steps and allow users to normalize the date as part of a single grok processor instead of forcing them to do this dance manually?

I guess there are still cases where you want to dance the dance, but then it's important to allow the user to clean up the temporary time field which we don't do right now. Should we add the drop field processor that to the list of processors we want to support? Seems like that one is helpful also to cut down storage costs.

[Kerry350]

@flash1293

Some more general thoughts - wdyt about this @LucaWintergerst and @Kerry350 ? [...]

Yeah, this seems like a good suggestion to me. But I think we should focus on getting the core functionality in first, before adding these (helpful) additional steps.

Should we add the drop field processor that to the list of processors we want to support? Seems like that one is helpful also to cut down storage costs.

This seems like a good first step, so at least cleanup can be carried out.

Then we could almost look at adding something like "workflows" (just a random name, could be called anything) that wrap multiple processor steps in a graceful way. We could maybe add these for common, known workflows (like this).

[LucaWintergerst]

good suggestion @flash1293 , I agree that in the coming weeks we should work towards more workflows of some kind.
It could be as simple as during the grok step, selecting the field as the "main" timestamp field and we'll just take are of the rest and worst case ask for confirmation of the timestamp pattern (although since we have the GROK expression we should be able to infer that with super high accuracy)
under the hood we'd then do the dance, and hide it from the UI

We'll need to think this through when the right time for this is - I think we should add support for all processors first, and then work on making the rough edges nicer

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0efed29

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
streamsApp	388	391	+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/streams-schema	291	303	+12

Any counts in public APIs

Total count of every any typed public API. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats any for more detailed information.

id	before	after	diff
@kbn/streams-schema	6	7	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
streamsApp	370.8KB	375.5KB	+4.7KB

Unknown metric groups

API count

id	before	after	diff
@kbn/streams-schema	295	307	+12

History

• 💚 Build #285508 succeeded f0f49fb
• 💔 Build #285035 failed cc4c4be
• 💚 Build #283871 succeeded 1007137
• 💛 Build #283219 was flaky 4152584
• 💚 Build #281814 succeeded 0360242
• 💔 Build #281794 failed 78e5fdb

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13969868191

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 213559

Questions ?

Please refer to the Backport tool documentation

[flash1293]

Oh for sure, that was all future talk but I put it here because of the context :) Agreed with adding the processors like this for now, I don't think that use case will go away.

[Kerry350]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[tonyghiani]

Just followed the discussion and changes and everything LGTM, thanks for the carrying this on @Kerry350!