[Streams] Grok UI integration with enrichment page#219146
[Streams] Grok UI integration with enrichment page#219146Kerry350 merged 14 commits intoelastic:mainfrom
Conversation
588fd86 to
a95b542
Compare
|
0d09449 to
253f11f
Compare
c9c27bc to
0449765
Compare
tonyghiani
left a comment
There was a problem hiding this comment.
I tried to review the PR in as much detail as possible to provide a minor second round of feedback. While testing the app, I found some functional issues:
- It seems these changes no longer display the source field and parsed fields as before. I assume this is because the highlight feature is now considered sufficient, but there are edge cases where viewing the resulting fields is still necessary.
The recording demonstrates what happens when a user overrides the message (or any other source field) after parsing some content. In this case, only the remaining content is displayed, with no insights into the parsed fields directly in the table, nor is the highlight shown.
IMO, we should keep the columns with the parsed results. This not only provides visibility into what the parsed fields are, but also makes it easier to catch field name typos (e.g., %{LOGLEVEL:log.levl}-missing the "e").
Screen.Recording.2025-05-09.at.11.09.13.mov
- When no field is defined in the processor config, the table should display the complete previewDocuments. Currently, however, it shows an empty grid.
Screen.Recording.2025-05-09.at.11.15.07.mov
src/platform/packages/shared/kbn-grok-ui/components/expression.tsx
Outdated
Show resolved
Hide resolved
src/platform/packages/shared/kbn-grok-ui/components/sample_input.tsx
Outdated
Show resolved
Hide resolved
src/platform/packages/shared/kbn-grok-ui/components/sample_input.tsx
Outdated
Show resolved
Hide resolved
src/platform/packages/shared/kbn-grok-ui/components/sample_input.tsx
Outdated
Show resolved
Hide resolved
| const renderProcessedSample = (draftGrokExpressions: DraftGrokExpression[], sample: string) => { | ||
| for (const expression of draftGrokExpressions) { | ||
| const regexp = expression.getRegex(); | ||
| if (regexp) { | ||
| const result = sample?.match(regexp); | ||
| if (result !== null) { | ||
| const regexpPatternSource = expression.getRegex(); | ||
| const fields = expression.getFields(); | ||
|
|
||
| if (regexpPatternSource) { | ||
| const regexpPattern = new RegExp( | ||
| regexpPatternSource.source, | ||
| // d flag is added to allow for indices tracking | ||
| regexpPatternSource.flags + 'd' | ||
| ); | ||
|
|
||
| // We expect one match per sample (we are not using global matches / flags) or none | ||
| const match = sample.match(regexpPattern); | ||
|
|
||
| const highlightSpans = []; | ||
|
|
||
| // Overall continuous match highlight (highlights the whole Grok pattern match) | ||
| if (match && match.length > 0) { | ||
| const matchingText = match[0]; | ||
| const startIndex = match.index; | ||
|
|
||
| if (startIndex !== undefined) { | ||
| const endIndex = startIndex + matchingText.length; | ||
| highlightSpans.push({ startIndex, endIndex, className: 'grok-pattern-match' }); | ||
| } | ||
| } | ||
|
|
||
| // Semantic (field name) match highlights | ||
| const matchGroupResults = regexpPattern.exec(sample); | ||
| if (matchGroupResults && matchGroupResults.indices && matchGroupResults.indices.groups) { | ||
| for (const [key, value] of Object.entries(matchGroupResults.indices.groups)) { | ||
| const fieldDefinition = fields.get(key); | ||
| if (value && fieldDefinition) { | ||
| const [startIndex, endIndex] = value; | ||
| highlightSpans.push({ | ||
| startIndex, | ||
| endIndex, | ||
| className: colourToClassName(fieldDefinition?.colour), | ||
| fieldDefinition, | ||
| }); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Apply highlights to the sample string | ||
| const highlightedSample = applyHighlights(sample, highlightSpans); | ||
|
|
||
| return <div>{highlightedSample}</div>; | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return <>{sample}</>; | ||
| }; |
There was a problem hiding this comment.
note(non-blocking): it seems there are some shared pure parts between this logic and the one defined in for the sample input component, do you think we could extract some of those pieces into reusable and more descriptive utils?
For instance, I'm referring to parts like creating/matching the regexp pattern or assertions for existing matches.
...latform/plugins/shared/streams_app/public/components/data_management/preview_table/index.tsx
Outdated
Show resolved
Hide resolved
...latform/plugins/shared/streams_app/public/components/data_management/preview_table/index.tsx
Outdated
Show resolved
Hide resolved
...app/public/components/data_management/stream_detail_enrichment/processor_outcome_preview.tsx
Outdated
Show resolved
Hide resolved
...ared/streams_app/public/components/data_management/stream_detail_enrichment/page_content.tsx
Outdated
Show resolved
Hide resolved
...ponents/data_management/stream_detail_enrichment/processors/grok/grok_custom_sample_data.tsx
Outdated
Show resolved
Hide resolved
|
@tonyghiani Thanks so much for the review, gives me something to focus on when I have FOMO next week 😂 Regarding your two functionality comments:
This was modelled off the designs, which had explicit bits of text around this (see the yellow text). I can see pros and cons to both (as in, this tries to give as much space as possible to the highlighting etc). @patpscal What do you think about Marco's proposal?
This is actually a bug on |
8ae2794 to
e662a4a
Compare
…ell here) Only render preview with Grok functionality when field is selected
|
@flash1293 Thanks for looking.
This is fixed 👍 When I refactored the debouncing logic I forgot one of the dependencies in the
This is also fixed 👍 This could potentially have further performance improvements but I'd like to defer that as it'd involve changing things higher in the hierarchy around the data grid, virtualisation / cell rendering etc. At the moment (at least in my local testing) there's no perceived performance issues. @tonyghiani @flash1293 I'd really, really like to get this merged as it's a large PR. There was an open question around the columns, but I think we're okay with moving forward with this (e.g. matching the designs)? The only outstanding feedback was this: #219146 (comment), that one I'd like to defer just so I can add the helpers / unit tests to the package without expanding this PR further. |
There was a problem hiding this comment.
This looks mostly good to me, there is one case I'm not sure about.
But when I replace the message with a subset of the original message (which is a very common thing to do), then this happens:

it basically doesn't show me anymore what got parsed into different places. I think we should fix this case - either by not changing the logic for displayed columns just yet or by handling this case in a special way (if the message gets overwritten in the pattern, show all the columns, otherwise just show the message column). This will buy us some time to fix it in an even better way later on (e.g. we keep showing the original raw message with the highlight instead of overwriting with the new message), but I don't think we should regress the experience in this important case.
Does that make sense? Otherwise it's good to go from my side
💚 Build Succeeded
Metrics [docs]Module Count
Public APIs missing comments
Async chunks
Page load bundle
History
cc @Kerry350 |
|
++ regarding the issue Joe has highlighted, I noticed that too on my first PR review, the rest looks good 👌 |
|
@flash1293 I've pushed a change that addresses your concerns above. If a Grok pattern attempts to overwrite the configured field (e.g. a semantic name that matches the field) then the standard preview table (with all columns) is used. The highlight is removed, since we'd be applying the full pattern back to a "partial" (at this stage) simulated / parsed field in the document.
This will be the good long term solution. I think we'll need to think through this though as it depends what exactly we mean by original, it's not just a case of grabbing the true original from the sample document since we might have used other processors to create this Grok processors' field, so it's more a case of "original at the point of entering the simulation of this processor" 🤔 (if that makes sense). |
|
Starting backport for target branches: 8.19 https://github.com/elastic/kibana/actions/runs/15319345578 |
## Summary Closes elastic/streams-program#173. From a high level this takes the UI components / models in the Grok UI package from a POC level to a polished level, and then integrates those with the Enrichment / Extraction page in the Streams UI. The Grok UI package is completely agnostic to it's consumer, so these package resources should be ready to integrate with any part of Kibana (we want to eventually bring this to Discover, for example). ## Notes for the reviewer Quite a few notes, but hopefully they help navigate the code and decisions made. - The `PreviewTable` now allows a custom `renderCellValue` prop, this is used when we are dealing with a **draft** Grok processor. - With processors we have a saved state (these don't run in the simulation), a staged state (configured and added but not saved, these are used in the simulation), and a draft state (freshly added, but not staged / saved yet, these are used in the simulation). The Grok highlighting is only available in the simulation table when the Grok processor if in a **draft** state, this is because we can't cleanly differentiate between staged processors at the moment which one exactly is being edited (multiple edit panels can be opened at once). I have discussed this with @tonyghiani and @patpscal and we've agreed this is fine to start with, with elastic#218955 and moving to only one edit at a time we will be able to roll this out to editing staged processors as well as the draft processor. - Custom sample data is **always** available, so there is always a way to test the patterns with highlights etc. - In the Grok UI package there is a read only version of the sample component, this is for things like cell rendering within a table and provides it's tooltips via EUI. There is also an "input" version which is backed by Monaco (tooltips also provided by Monaco). The styling differs on the tooltips a bit between the two for this reason, I will try and enhance / override the Monaco styles so they adhere to our colour palette / themes etc in a followup. - Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint to fetch the patterns, however I couldn't find a convenient way to do this yet without adding an API endpoint somewhere (since this needs an ES client). I've deferred this for now, but it's why I've preempted `setup()` being modelled as `async`. - I had to pass `formControl` around in a bit of an ugly way in the patterns editor, this was purely due to the `useForm()` hook not working as I expected in that part of the hierarchy. - We only need a single `GrokCollection` instance, as such this is stored in the top level state machine. - We technically have two areas where state is managed. We have the state machines and then we have the form state managed by React Hook Form. The form state is synchronised back to the state machines when processor changes are made. This Grok integration work introduces a new need which is for resources to be shared between the processors panel and the simulation panel, in this case the same `DraftGrokExpression` instances. There are multiple ways this could have been modelled, with various pros and cons, but I wanted to keep the paradigm we have now which is that the processor definitions in the state machine are "complete" (in the sense they're converted) and not partial form state. The `DraftGrokExpressions` are integrated directly with the form state, and synchronised back to a new `resources` property in the state machine. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> (cherry picked from commit 216ac7e)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
## Summary Closes elastic/streams-program#173. From a high level this takes the UI components / models in the Grok UI package from a POC level to a polished level, and then integrates those with the Enrichment / Extraction page in the Streams UI. The Grok UI package is completely agnostic to it's consumer, so these package resources should be ready to integrate with any part of Kibana (we want to eventually bring this to Discover, for example). ## Notes for the reviewer Quite a few notes, but hopefully they help navigate the code and decisions made. - The `PreviewTable` now allows a custom `renderCellValue` prop, this is used when we are dealing with a **draft** Grok processor. - With processors we have a saved state (these don't run in the simulation), a staged state (configured and added but not saved, these are used in the simulation), and a draft state (freshly added, but not staged / saved yet, these are used in the simulation). The Grok highlighting is only available in the simulation table when the Grok processor if in a **draft** state, this is because we can't cleanly differentiate between staged processors at the moment which one exactly is being edited (multiple edit panels can be opened at once). I have discussed this with @tonyghiani and @patpscal and we've agreed this is fine to start with, with elastic#218955 and moving to only one edit at a time we will be able to roll this out to editing staged processors as well as the draft processor. - Custom sample data is **always** available, so there is always a way to test the patterns with highlights etc. - In the Grok UI package there is a read only version of the sample component, this is for things like cell rendering within a table and provides it's tooltips via EUI. There is also an "input" version which is backed by Monaco (tooltips also provided by Monaco). The styling differs on the tooltips a bit between the two for this reason, I will try and enhance / override the Monaco styles so they adhere to our colour palette / themes etc in a followup. - Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint to fetch the patterns, however I couldn't find a convenient way to do this yet without adding an API endpoint somewhere (since this needs an ES client). I've deferred this for now, but it's why I've preempted `setup()` being modelled as `async`. - I had to pass `formControl` around in a bit of an ugly way in the patterns editor, this was purely due to the `useForm()` hook not working as I expected in that part of the hierarchy. - We only need a single `GrokCollection` instance, as such this is stored in the top level state machine. - We technically have two areas where state is managed. We have the state machines and then we have the form state managed by React Hook Form. The form state is synchronised back to the state machines when processor changes are made. This Grok integration work introduces a new need which is for resources to be shared between the processors panel and the simulation panel, in this case the same `DraftGrokExpression` instances. There are multiple ways this could have been modelled, with various pros and cons, but I wanted to keep the paradigm we have now which is that the processor definitions in the state machine are "complete" (in the sense they're converted) and not partial form state. The `DraftGrokExpressions` are integrated directly with the form state, and synchronised back to a new `resources` property in the state machine. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
…221887) # Backport This will backport the following commits from `main` to `8.19`: - [[Streams] Grok UI integration with enrichment page (#219146)](#219146) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Kerry Gallagher","email":"kerry.gallagher@elastic.co"},"sourceCommit":{"committedDate":"2025-05-29T08:07:31Z","message":"[Streams] Grok UI integration with enrichment page (#219146)\n\n## Summary\n\nCloses https://github.com/elastic/streams-program/issues/173.\n\nFrom a high level this takes the UI components / models in the Grok UI\npackage from a POC level to a polished level, and then integrates those\nwith the Enrichment / Extraction page in the Streams UI. The Grok UI\npackage is completely agnostic to it's consumer, so these package\nresources should be ready to integrate with any part of Kibana (we want\nto eventually bring this to Discover, for example).\n\n## Notes for the reviewer\n\nQuite a few notes, but hopefully they help navigate the code and\ndecisions made.\n\n- The `PreviewTable` now allows a custom `renderCellValue` prop, this is\nused when we are dealing with a **draft** Grok processor.\n\n- With processors we have a saved state (these don't run in the\nsimulation), a staged state (configured and added but not saved, these\nare used in the simulation), and a draft state (freshly added, but not\nstaged / saved yet, these are used in the simulation). The Grok\nhighlighting is only available in the simulation table when the Grok\nprocessor if in a **draft** state, this is because we can't cleanly\ndifferentiate between staged processors at the moment which one exactly\nis being edited (multiple edit panels can be opened at once). I have\ndiscussed this with @tonyghiani and @patpscal and we've agreed this is\nfine to start with, with\nhttps://github.com//issues/218955?reload=1?reload=1 and\nmoving to only one edit at a time we will be able to roll this out to\nediting staged processors as well as the draft processor.\n\n- Custom sample data is **always** available, so there is always a way\nto test the patterns with highlights etc.\n\n- In the Grok UI package there is a read only version of the sample\ncomponent, this is for things like cell rendering within a table and\nprovides it's tooltips via EUI. There is also an \"input\" version which\nis backed by Monaco (tooltips also provided by Monaco). The styling\ndiffers on the tooltips a bit between the two for this reason, I will\ntry and enhance / override the Monaco styles so they adhere to our\ncolour palette / themes etc in a followup.\n\n- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint\nto fetch the patterns, however I couldn't find a convenient way to do\nthis yet without adding an API endpoint somewhere (since this needs an\nES client). I've deferred this for now, but it's why I've preempted\n`setup()` being modelled as `async`.\n\n- I had to pass `formControl` around in a bit of an ugly way in the\npatterns editor, this was purely due to the `useForm()` hook not working\nas I expected in that part of the hierarchy.\n\n- We only need a single `GrokCollection` instance, as such this is\nstored in the top level state machine.\n\n- We technically have two areas where state is managed. We have the\nstate machines and then we have the form state managed by React Hook\nForm. The form state is synchronised back to the state machines when\nprocessor changes are made. This Grok integration work introduces a new\nneed which is for resources to be shared between the processors panel\nand the simulation panel, in this case the same `DraftGrokExpression`\ninstances. There are multiple ways this could have been modelled, with\nvarious pros and cons, but I wanted to keep the paradigm we have now\nwhich is that the processor definitions in the state machine are\n\"complete\" (in the sense they're converted) and not partial form state.\nThe `DraftGrokExpressions` are integrated directly with the form state,\nand synchronised back to a new `resources` property in the state\nmachine.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Joe Reuter <johannes.reuter@elastic.co>","sha":"216ac7e83eb4edb19482b3e0112fe64c4204a938","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","Feature:Streams","v9.1.0","v8.19.0"],"title":"[Streams] Grok UI integration with enrichment page","number":219146,"url":"https://github.com/elastic/kibana/pull/219146","mergeCommit":{"message":"[Streams] Grok UI integration with enrichment page (#219146)\n\n## Summary\n\nCloses https://github.com/elastic/streams-program/issues/173.\n\nFrom a high level this takes the UI components / models in the Grok UI\npackage from a POC level to a polished level, and then integrates those\nwith the Enrichment / Extraction page in the Streams UI. The Grok UI\npackage is completely agnostic to it's consumer, so these package\nresources should be ready to integrate with any part of Kibana (we want\nto eventually bring this to Discover, for example).\n\n## Notes for the reviewer\n\nQuite a few notes, but hopefully they help navigate the code and\ndecisions made.\n\n- The `PreviewTable` now allows a custom `renderCellValue` prop, this is\nused when we are dealing with a **draft** Grok processor.\n\n- With processors we have a saved state (these don't run in the\nsimulation), a staged state (configured and added but not saved, these\nare used in the simulation), and a draft state (freshly added, but not\nstaged / saved yet, these are used in the simulation). The Grok\nhighlighting is only available in the simulation table when the Grok\nprocessor if in a **draft** state, this is because we can't cleanly\ndifferentiate between staged processors at the moment which one exactly\nis being edited (multiple edit panels can be opened at once). I have\ndiscussed this with @tonyghiani and @patpscal and we've agreed this is\nfine to start with, with\nhttps://github.com//issues/218955?reload=1?reload=1 and\nmoving to only one edit at a time we will be able to roll this out to\nediting staged processors as well as the draft processor.\n\n- Custom sample data is **always** available, so there is always a way\nto test the patterns with highlights etc.\n\n- In the Grok UI package there is a read only version of the sample\ncomponent, this is for things like cell rendering within a table and\nprovides it's tooltips via EUI. There is also an \"input\" version which\nis backed by Monaco (tooltips also provided by Monaco). The styling\ndiffers on the tooltips a bit between the two for this reason, I will\ntry and enhance / override the Monaco styles so they adhere to our\ncolour palette / themes etc in a followup.\n\n- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint\nto fetch the patterns, however I couldn't find a convenient way to do\nthis yet without adding an API endpoint somewhere (since this needs an\nES client). I've deferred this for now, but it's why I've preempted\n`setup()` being modelled as `async`.\n\n- I had to pass `formControl` around in a bit of an ugly way in the\npatterns editor, this was purely due to the `useForm()` hook not working\nas I expected in that part of the hierarchy.\n\n- We only need a single `GrokCollection` instance, as such this is\nstored in the top level state machine.\n\n- We technically have two areas where state is managed. We have the\nstate machines and then we have the form state managed by React Hook\nForm. The form state is synchronised back to the state machines when\nprocessor changes are made. This Grok integration work introduces a new\nneed which is for resources to be shared between the processors panel\nand the simulation panel, in this case the same `DraftGrokExpression`\ninstances. There are multiple ways this could have been modelled, with\nvarious pros and cons, but I wanted to keep the paradigm we have now\nwhich is that the processor definitions in the state machine are\n\"complete\" (in the sense they're converted) and not partial form state.\nThe `DraftGrokExpressions` are integrated directly with the form state,\nand synchronised back to a new `resources` property in the state\nmachine.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Joe Reuter <johannes.reuter@elastic.co>","sha":"216ac7e83eb4edb19482b3e0112fe64c4204a938"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219146","number":219146,"mergeCommit":{"message":"[Streams] Grok UI integration with enrichment page (#219146)\n\n## Summary\n\nCloses https://github.com/elastic/streams-program/issues/173.\n\nFrom a high level this takes the UI components / models in the Grok UI\npackage from a POC level to a polished level, and then integrates those\nwith the Enrichment / Extraction page in the Streams UI. The Grok UI\npackage is completely agnostic to it's consumer, so these package\nresources should be ready to integrate with any part of Kibana (we want\nto eventually bring this to Discover, for example).\n\n## Notes for the reviewer\n\nQuite a few notes, but hopefully they help navigate the code and\ndecisions made.\n\n- The `PreviewTable` now allows a custom `renderCellValue` prop, this is\nused when we are dealing with a **draft** Grok processor.\n\n- With processors we have a saved state (these don't run in the\nsimulation), a staged state (configured and added but not saved, these\nare used in the simulation), and a draft state (freshly added, but not\nstaged / saved yet, these are used in the simulation). The Grok\nhighlighting is only available in the simulation table when the Grok\nprocessor if in a **draft** state, this is because we can't cleanly\ndifferentiate between staged processors at the moment which one exactly\nis being edited (multiple edit panels can be opened at once). I have\ndiscussed this with @tonyghiani and @patpscal and we've agreed this is\nfine to start with, with\nhttps://github.com//issues/218955?reload=1?reload=1 and\nmoving to only one edit at a time we will be able to roll this out to\nediting staged processors as well as the draft processor.\n\n- Custom sample data is **always** available, so there is always a way\nto test the patterns with highlights etc.\n\n- In the Grok UI package there is a read only version of the sample\ncomponent, this is for things like cell rendering within a table and\nprovides it's tooltips via EUI. There is also an \"input\" version which\nis backed by Monaco (tooltips also provided by Monaco). The styling\ndiffers on the tooltips a bit between the two for this reason, I will\ntry and enhance / override the Monaco styles so they adhere to our\ncolour palette / themes etc in a followup.\n\n- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint\nto fetch the patterns, however I couldn't find a convenient way to do\nthis yet without adding an API endpoint somewhere (since this needs an\nES client). I've deferred this for now, but it's why I've preempted\n`setup()` being modelled as `async`.\n\n- I had to pass `formControl` around in a bit of an ugly way in the\npatterns editor, this was purely due to the `useForm()` hook not working\nas I expected in that part of the hierarchy.\n\n- We only need a single `GrokCollection` instance, as such this is\nstored in the top level state machine.\n\n- We technically have two areas where state is managed. We have the\nstate machines and then we have the form state managed by React Hook\nForm. The form state is synchronised back to the state machines when\nprocessor changes are made. This Grok integration work introduces a new\nneed which is for resources to be shared between the processors panel\nand the simulation panel, in this case the same `DraftGrokExpression`\ninstances. There are multiple ways this could have been modelled, with\nvarious pros and cons, but I wanted to keep the paradigm we have now\nwhich is that the processor definitions in the state machine are\n\"complete\" (in the sense they're converted) and not partial form state.\nThe `DraftGrokExpressions` are integrated directly with the form state,\nand synchronised back to a new `resources` property in the state\nmachine.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Joe Reuter <johannes.reuter@elastic.co>","sha":"216ac7e83eb4edb19482b3e0112fe64c4204a938"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: Kerry Gallagher <kerry.gallagher@elastic.co> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com>
## Summary Closes elastic/streams-program#173. From a high level this takes the UI components / models in the Grok UI package from a POC level to a polished level, and then integrates those with the Enrichment / Extraction page in the Streams UI. The Grok UI package is completely agnostic to it's consumer, so these package resources should be ready to integrate with any part of Kibana (we want to eventually bring this to Discover, for example). ## Notes for the reviewer Quite a few notes, but hopefully they help navigate the code and decisions made. - The `PreviewTable` now allows a custom `renderCellValue` prop, this is used when we are dealing with a **draft** Grok processor. - With processors we have a saved state (these don't run in the simulation), a staged state (configured and added but not saved, these are used in the simulation), and a draft state (freshly added, but not staged / saved yet, these are used in the simulation). The Grok highlighting is only available in the simulation table when the Grok processor if in a **draft** state, this is because we can't cleanly differentiate between staged processors at the moment which one exactly is being edited (multiple edit panels can be opened at once). I have discussed this with @tonyghiani and @patpscal and we've agreed this is fine to start with, with elastic#218955 and moving to only one edit at a time we will be able to roll this out to editing staged processors as well as the draft processor. - Custom sample data is **always** available, so there is always a way to test the patterns with highlights etc. - In the Grok UI package there is a read only version of the sample component, this is for things like cell rendering within a table and provides it's tooltips via EUI. There is also an "input" version which is backed by Monaco (tooltips also provided by Monaco). The styling differs on the tooltips a bit between the two for this reason, I will try and enhance / override the Monaco styles so they adhere to our colour palette / themes etc in a followup. - Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint to fetch the patterns, however I couldn't find a convenient way to do this yet without adding an API endpoint somewhere (since this needs an ES client). I've deferred this for now, but it's why I've preempted `setup()` being modelled as `async`. - I had to pass `formControl` around in a bit of an ugly way in the patterns editor, this was purely due to the `useForm()` hook not working as I expected in that part of the hierarchy. - We only need a single `GrokCollection` instance, as such this is stored in the top level state machine. - We technically have two areas where state is managed. We have the state machines and then we have the form state managed by React Hook Form. The form state is synchronised back to the state machines when processor changes are made. This Grok integration work introduces a new need which is for resources to be shared between the processors panel and the simulation panel, in this case the same `DraftGrokExpression` instances. There are multiple ways this could have been modelled, with various pros and cons, but I wanted to keep the paradigm we have now which is that the processor definitions in the state machine are "complete" (in the sense they're converted) and not partial form state. The `DraftGrokExpressions` are integrated directly with the form state, and synchronised back to a new `resources` property in the state machine. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
## Summary Closes elastic/streams-program#173. From a high level this takes the UI components / models in the Grok UI package from a POC level to a polished level, and then integrates those with the Enrichment / Extraction page in the Streams UI. The Grok UI package is completely agnostic to it's consumer, so these package resources should be ready to integrate with any part of Kibana (we want to eventually bring this to Discover, for example). ## Notes for the reviewer Quite a few notes, but hopefully they help navigate the code and decisions made. - The `PreviewTable` now allows a custom `renderCellValue` prop, this is used when we are dealing with a **draft** Grok processor. - With processors we have a saved state (these don't run in the simulation), a staged state (configured and added but not saved, these are used in the simulation), and a draft state (freshly added, but not staged / saved yet, these are used in the simulation). The Grok highlighting is only available in the simulation table when the Grok processor if in a **draft** state, this is because we can't cleanly differentiate between staged processors at the moment which one exactly is being edited (multiple edit panels can be opened at once). I have discussed this with @tonyghiani and @patpscal and we've agreed this is fine to start with, with elastic#218955 and moving to only one edit at a time we will be able to roll this out to editing staged processors as well as the draft processor. - Custom sample data is **always** available, so there is always a way to test the patterns with highlights etc. - In the Grok UI package there is a read only version of the sample component, this is for things like cell rendering within a table and provides it's tooltips via EUI. There is also an "input" version which is backed by Monaco (tooltips also provided by Monaco). The styling differs on the tooltips a bit between the two for this reason, I will try and enhance / override the Monaco styles so they adhere to our colour palette / themes etc in a followup. - Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint to fetch the patterns, however I couldn't find a convenient way to do this yet without adding an API endpoint somewhere (since this needs an ES client). I've deferred this for now, but it's why I've preempted `setup()` being modelled as `async`. - I had to pass `formControl` around in a bit of an ugly way in the patterns editor, this was purely due to the `useForm()` hook not working as I expected in that part of the hierarchy. - We only need a single `GrokCollection` instance, as such this is stored in the top level state machine. - We technically have two areas where state is managed. We have the state machines and then we have the form state managed by React Hook Form. The form state is synchronised back to the state machines when processor changes are made. This Grok integration work introduces a new need which is for resources to be shared between the processors panel and the simulation panel, in this case the same `DraftGrokExpression` instances. There are multiple ways this could have been modelled, with various pros and cons, but I wanted to keep the paradigm we have now which is that the processor definitions in the state machine are "complete" (in the sense they're converted) and not partial form state. The `DraftGrokExpressions` are integrated directly with the form state, and synchronised back to a new `resources` property in the state machine. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>






Summary
Closes https://github.com/elastic/streams-program/issues/173.
From a high level this takes the UI components / models in the Grok UI package from a POC level to a polished level, and then integrates those with the Enrichment / Extraction page in the Streams UI. The Grok UI package is completely agnostic to it's consumer, so these package resources should be ready to integrate with any part of Kibana (we want to eventually bring this to Discover, for example).
Notes for the reviewer
Quite a few notes, but hopefully they help navigate the code and decisions made.
The
PreviewTablenow allows a customrenderCellValueprop, this is used when we are dealing with a draft Grok processor.With processors we have a saved state (these don't run in the simulation), a staged state (configured and added but not saved, these are used in the simulation), and a draft state (freshly added, but not staged / saved yet, these are used in the simulation). The Grok highlighting is only available in the simulation table when the Grok processor if in a draft state, this is because we can't cleanly differentiate between staged processors at the moment which one exactly is being edited (multiple edit panels can be opened at once). I have discussed this with @tonyghiani and @patpscal and we've agreed this is fine to start with, with [Streams 🌊] Build processing form only when a processor is under edit #218955 and moving to only one edit at a time we will be able to roll this out to editing staged processors as well as the draft processor.
In the Grok UI package there is a read only version of the sample component, this is for things like cell rendering within a table and provides it's tooltips via EUI. There is also an "input" version which is backed by Monaco (tooltips also provided by Monaco). The styling differs on the tooltips a bit between the two for this reason, I will try and enhance / override the Monaco styles so they adhere to our colour palette / themes etc in a followup.
Giorgos had suggested using the
/_ingest/processor/grokES endpoint to fetch the patterns, however I couldn't find a convenient way to do this yet without adding an API endpoint somewhere (since this needs an ES client). I've deferred this for now, but it's why I've preemptedsetup()being modelled asasync.I had to pass
formControlaround in a bit of an ugly way in the patterns editor, this was purely due to theuseForm()hook not working as I expected in that part of the hierarchy.We only need a single
GrokCollectioninstance, as such this is stored in the top level state machine.We technically have two areas where state is managed. We have the state machines and then we have the form state managed by React Hook Form. The form state is synchronised back to the state machines when processor changes are made. This Grok integration work introduces a new need which is for resources to be shared between the processors panel and the simulation panel, in this case the same
DraftGrokExpressioninstances. There are multiple ways this could have been modelled, with various pros and cons, but I wanted to keep the paradigm we have now which is that the processor definitions in the state machine are "complete" (in the sense they're converted) and not partial form state. TheDraftGrokExpressionsare integrated directly with the form state, and synchronised back to a newresourcesproperty in the state machine.