Skip to content

[Streams] Grok UI integration with enrichment page#219146

Merged
Kerry350 merged 14 commits intoelastic:mainfrom
Kerry350:173-grok-ui-integration
May 29, 2025
Merged

[Streams] Grok UI integration with enrichment page#219146
Kerry350 merged 14 commits intoelastic:mainfrom
Kerry350:173-grok-ui-integration

Conversation

@Kerry350
Copy link
Contributor

@Kerry350 Kerry350 commented Apr 24, 2025

Summary

Closes https://github.com/elastic/streams-program/issues/173.

From a high level this takes the UI components / models in the Grok UI package from a POC level to a polished level, and then integrates those with the Enrichment / Extraction page in the Streams UI. The Grok UI package is completely agnostic to it's consumer, so these package resources should be ready to integrate with any part of Kibana (we want to eventually bring this to Discover, for example).

Notes for the reviewer

Quite a few notes, but hopefully they help navigate the code and decisions made.

  • The PreviewTable now allows a custom renderCellValue prop, this is used when we are dealing with a draft Grok processor.

  • With processors we have a saved state (these don't run in the simulation), a staged state (configured and added but not saved, these are used in the simulation), and a draft state (freshly added, but not staged / saved yet, these are used in the simulation). The Grok highlighting is only available in the simulation table when the Grok processor if in a draft state, this is because we can't cleanly differentiate between staged processors at the moment which one exactly is being edited (multiple edit panels can be opened at once). I have discussed this with @tonyghiani and @patpscal and we've agreed this is fine to start with, with [Streams 🌊] Build processing form only when a processor is under edit #218955 and moving to only one edit at a time we will be able to roll this out to editing staged processors as well as the draft processor.

    • Custom sample data is always available, so there is always a way to test the patterns with highlights etc.
  • In the Grok UI package there is a read only version of the sample component, this is for things like cell rendering within a table and provides it's tooltips via EUI. There is also an "input" version which is backed by Monaco (tooltips also provided by Monaco). The styling differs on the tooltips a bit between the two for this reason, I will try and enhance / override the Monaco styles so they adhere to our colour palette / themes etc in a followup.

  • Giorgos had suggested using the /_ingest/processor/grok ES endpoint to fetch the patterns, however I couldn't find a convenient way to do this yet without adding an API endpoint somewhere (since this needs an ES client). I've deferred this for now, but it's why I've preempted setup() being modelled as async.

  • I had to pass formControl around in a bit of an ugly way in the patterns editor, this was purely due to the useForm() hook not working as I expected in that part of the hierarchy.

  • We only need a single GrokCollection instance, as such this is stored in the top level state machine.

  • We technically have two areas where state is managed. We have the state machines and then we have the form state managed by React Hook Form. The form state is synchronised back to the state machines when processor changes are made. This Grok integration work introduces a new need which is for resources to be shared between the processors panel and the simulation panel, in this case the same DraftGrokExpression instances. There are multiple ways this could have been modelled, with various pros and cons, but I wanted to keep the paradigm we have now which is that the processor definitions in the state machine are "complete" (in the sense they're converted) and not partial form state. The DraftGrokExpressions are integrated directly with the form state, and synchronised back to a new resources property in the state machine.

@Kerry350 Kerry350 self-assigned this Apr 24, 2025
@Kerry350 Kerry350 force-pushed the 173-grok-ui-integration branch 3 times, most recently from 588fd86 to a95b542 Compare May 2, 2025 08:16
@prodsecmachine
Copy link
Collaborator

prodsecmachine commented May 2, 2025

⚠️ Snyk checks are incomplete.

⚠️ security/snyk check encountered an error. (View Details)

⚠️ license/snyk check encountered an error. (View Details)

@Kerry350 Kerry350 force-pushed the 173-grok-ui-integration branch 4 times, most recently from 0d09449 to 253f11f Compare May 8, 2025 13:25
@Kerry350 Kerry350 added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels v9.1.0 v8.19.0 Feature:Streams This is the label for the Streams Project labels May 8, 2025
@Kerry350 Kerry350 requested a review from tonyghiani May 8, 2025 14:08
@Kerry350 Kerry350 marked this pull request as ready for review May 8, 2025 14:16
@Kerry350 Kerry350 requested review from a team as code owners May 8, 2025 14:16
@Kerry350 Kerry350 force-pushed the 173-grok-ui-integration branch from c9c27bc to 0449765 Compare May 8, 2025 14:51
Copy link
Contributor

@tonyghiani tonyghiani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to review the PR in as much detail as possible to provide a minor second round of feedback. While testing the app, I found some functional issues:

  • It seems these changes no longer display the source field and parsed fields as before. I assume this is because the highlight feature is now considered sufficient, but there are edge cases where viewing the resulting fields is still necessary.
    The recording demonstrates what happens when a user overrides the message (or any other source field) after parsing some content. In this case, only the remaining content is displayed, with no insights into the parsed fields directly in the table, nor is the highlight shown.
    IMO, we should keep the columns with the parsed results. This not only provides visibility into what the parsed fields are, but also makes it easier to catch field name typos (e.g., %{LOGLEVEL:log.levl}-missing the "e").
Screen.Recording.2025-05-09.at.11.09.13.mov
  • When no field is defined in the processor config, the table should display the complete previewDocuments. Currently, however, it shows an empty grid.
Screen.Recording.2025-05-09.at.11.15.07.mov

Comment on lines +52 to +123
const renderProcessedSample = (draftGrokExpressions: DraftGrokExpression[], sample: string) => {
for (const expression of draftGrokExpressions) {
const regexp = expression.getRegex();
if (regexp) {
const result = sample?.match(regexp);
if (result !== null) {
const regexpPatternSource = expression.getRegex();
const fields = expression.getFields();

if (regexpPatternSource) {
const regexpPattern = new RegExp(
regexpPatternSource.source,
// d flag is added to allow for indices tracking
regexpPatternSource.flags + 'd'
);

// We expect one match per sample (we are not using global matches / flags) or none
const match = sample.match(regexpPattern);

const highlightSpans = [];

// Overall continuous match highlight (highlights the whole Grok pattern match)
if (match && match.length > 0) {
const matchingText = match[0];
const startIndex = match.index;

if (startIndex !== undefined) {
const endIndex = startIndex + matchingText.length;
highlightSpans.push({ startIndex, endIndex, className: 'grok-pattern-match' });
}
}

// Semantic (field name) match highlights
const matchGroupResults = regexpPattern.exec(sample);
if (matchGroupResults && matchGroupResults.indices && matchGroupResults.indices.groups) {
for (const [key, value] of Object.entries(matchGroupResults.indices.groups)) {
const fieldDefinition = fields.get(key);
if (value && fieldDefinition) {
const [startIndex, endIndex] = value;
highlightSpans.push({
startIndex,
endIndex,
className: colourToClassName(fieldDefinition?.colour),
fieldDefinition,
});
}
}
}

// Apply highlights to the sample string
const highlightedSample = applyHighlights(sample, highlightSpans);

return <div>{highlightedSample}</div>;
}
}
}
}

return <>{sample}</>;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note(non-blocking): it seems there are some shared pure parts between this logic and the one defined in for the sample input component, do you think we could extract some of those pieces into reusable and more descriptive utils?
For instance, I'm referring to parts like creating/matching the regexp pattern or assertions for existing matches.

@Kerry350
Copy link
Contributor Author

Kerry350 commented May 9, 2025

@tonyghiani Thanks so much for the review, gives me something to focus on when I have FOMO next week 😂

Regarding your two functionality comments:

It seems these changes no longer display the source field and parsed fields as before. I assume this is because the highlight feature is now considered sufficient, but there are edge cases where viewing the resulting fields is still necessary.

This was modelled off the designs, which had explicit bits of text around this (see the yellow text).

Screenshot 2025-05-09 at 14 27 57

Screenshot 2025-05-09 at 14 28 06

I can see pros and cons to both (as in, this tries to give as much space as possible to the highlighting etc). @patpscal What do you think about Marco's proposal?

When no field is defined in the processor config, the table should display the complete previewDocuments. Currently, however, it shows an empty grid.

This is actually a bug on main, can be reproduced by just adding a Grok processor and removing the field there. I think it might have been introduced with Joe's "select best text field" work. I'll take a look at fixing it though 👍

@Kerry350 Kerry350 force-pushed the 173-grok-ui-integration branch from 8ae2794 to e662a4a Compare May 14, 2025 12:37
@flash1293
Copy link
Contributor

flash1293 commented May 23, 2025

Something weird seems to be going on with the custom sample data.

I'm copy/pasting the exact message from the right, but it just breaks:
Kapture 2025-05-23 at 17 02 51

It gets even worse when changing the pattern:
Kapture 2025-05-23 at 17 04 12

The other thing is that highlighting in the preview table is blinking while stuff is loading:
Kapture 2025-05-23 at 17 05 53

Is there a way to keep that stable? It makes sense to still run the simulation behind the scenes to learn about potential errors or so, but we don't show that output anyway. I tried using throttleTime instead of debounceTime for the Sample component, and it seems to improve the situation, but it still seems to rerender more often than necessary. Maybe we can have an LRU cache or something like that for messages based on the current pattern?

@Kerry350
Copy link
Contributor Author

@flash1293 Thanks for looking.

Something weird seems to be going on with the custom sample data.

This is fixed 👍 When I refactored the debouncing logic I forgot one of the dependencies in the useEffect dependency array.

The other thing is that highlighting in the preview table is blinking while stuff is loading:

This is also fixed 👍 This could potentially have further performance improvements but I'd like to defer that as it'd involve changing things higher in the hierarchy around the data grid, virtualisation / cell rendering etc. At the moment (at least in my local testing) there's no perceived performance issues.

@tonyghiani @flash1293 I'd really, really like to get this merged as it's a large PR.

There was an open question around the columns, but I think we're okay with moving forward with this (e.g. matching the designs)?

The only outstanding feedback was this: #219146 (comment), that one I'd like to defer just so I can add the helpers / unit tests to the package without expanding this PR further.

Copy link
Contributor

@flash1293 flash1293 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks mostly good to me, there is one case I'm not sure about.

This pattern is fine:
Screenshot 2025-05-27 at 17 12 27

But when I replace the message with a subset of the original message (which is a very common thing to do), then this happens:
Screenshot 2025-05-27 at 17 12 52

it basically doesn't show me anymore what got parsed into different places. I think we should fix this case - either by not changing the logic for displayed columns just yet or by handling this case in a special way (if the message gets overwritten in the pattern, show all the columns, otherwise just show the message column). This will buy us some time to fix it in an even better way later on (e.g. we keep showing the original raw message with the highlight instead of overwriting with the new message), but I don't think we should regress the experience in this important case.

Does that make sense? Otherwise it's good to go from my side

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
streamsApp 468 496 +28

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/grok-ui 2 71 +69

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
streamsApp 424.6KB 547.3KB +122.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
streamsApp 10.3KB 10.5KB +182.0B
Unknown metric groups

API count

id before after diff
@kbn/grok-ui 2 71 +69

History

cc @Kerry350

@tonyghiani
Copy link
Contributor

++ regarding the issue Joe has highlighted, I noticed that too on my first PR review, the rest looks good 👌

@Kerry350
Copy link
Contributor Author

@flash1293 I've pushed a change that addresses your concerns above. If a Grok pattern attempts to overwrite the configured field (e.g. a semantic name that matches the field) then the standard preview table (with all columns) is used. The highlight is removed, since we'd be applying the full pattern back to a "partial" (at this stage) simulated / parsed field in the document.

This will buy us some time to fix it in an even better way later on (e.g. we keep showing the original raw message with the highlight instead of overwriting with the new message)

This will be the good long term solution. I think we'll need to think through this though as it depends what exactly we mean by original, it's not just a case of grabbing the true original from the sample document since we might have used other processors to create this Grok processors' field, so it's more a case of "original at the point of entering the simulation of this processor" 🤔 (if that makes sense).

Copy link
Contributor

@flash1293 flash1293 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@Kerry350 Kerry350 merged commit 216ac7e into elastic:main May 29, 2025
10 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15319345578

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request May 29, 2025
## Summary

Closes elastic/streams-program#173.

From a high level this takes the UI components / models in the Grok UI
package from a POC level to a polished level, and then integrates those
with the Enrichment / Extraction page in the Streams UI. The Grok UI
package is completely agnostic to it's consumer, so these package
resources should be ready to integrate with any part of Kibana (we want
to eventually bring this to Discover, for example).

## Notes for the reviewer

Quite a few notes, but hopefully they help navigate the code and
decisions made.

- The `PreviewTable` now allows a custom `renderCellValue` prop, this is
used when we are dealing with a **draft** Grok processor.

- With processors we have a saved state (these don't run in the
simulation), a staged state (configured and added but not saved, these
are used in the simulation), and a draft state (freshly added, but not
staged / saved yet, these are used in the simulation). The Grok
highlighting is only available in the simulation table when the Grok
processor if in a **draft** state, this is because we can't cleanly
differentiate between staged processors at the moment which one exactly
is being edited (multiple edit panels can be opened at once). I have
discussed this with @tonyghiani and @patpscal and we've agreed this is
fine to start with, with
elastic#218955 and
moving to only one edit at a time we will be able to roll this out to
editing staged processors as well as the draft processor.

- Custom sample data is **always** available, so there is always a way
to test the patterns with highlights etc.

- In the Grok UI package there is a read only version of the sample
component, this is for things like cell rendering within a table and
provides it's tooltips via EUI. There is also an "input" version which
is backed by Monaco (tooltips also provided by Monaco). The styling
differs on the tooltips a bit between the two for this reason, I will
try and enhance / override the Monaco styles so they adhere to our
colour palette / themes etc in a followup.

- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint
to fetch the patterns, however I couldn't find a convenient way to do
this yet without adding an API endpoint somewhere (since this needs an
ES client). I've deferred this for now, but it's why I've preempted
`setup()` being modelled as `async`.

- I had to pass `formControl` around in a bit of an ugly way in the
patterns editor, this was purely due to the `useForm()` hook not working
as I expected in that part of the hierarchy.

- We only need a single `GrokCollection` instance, as such this is
stored in the top level state machine.

- We technically have two areas where state is managed. We have the
state machines and then we have the form state managed by React Hook
Form. The form state is synchronised back to the state machines when
processor changes are made. This Grok integration work introduces a new
need which is for resources to be shared between the processors panel
and the simulation panel, in this case the same `DraftGrokExpression`
instances. There are multiple ways this could have been modelled, with
various pros and cons, but I wanted to keep the paradigm we have now
which is that the processor definitions in the state machine are
"complete" (in the sense they're converted) and not partial form state.
The `DraftGrokExpressions` are integrated directly with the form state,
and synchronised back to a new `resources` property in the state
machine.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
(cherry picked from commit 216ac7e)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

akowalska622 pushed a commit to akowalska622/kibana that referenced this pull request May 29, 2025
## Summary

Closes elastic/streams-program#173.

From a high level this takes the UI components / models in the Grok UI
package from a POC level to a polished level, and then integrates those
with the Enrichment / Extraction page in the Streams UI. The Grok UI
package is completely agnostic to it's consumer, so these package
resources should be ready to integrate with any part of Kibana (we want
to eventually bring this to Discover, for example).

## Notes for the reviewer

Quite a few notes, but hopefully they help navigate the code and
decisions made.

- The `PreviewTable` now allows a custom `renderCellValue` prop, this is
used when we are dealing with a **draft** Grok processor.

- With processors we have a saved state (these don't run in the
simulation), a staged state (configured and added but not saved, these
are used in the simulation), and a draft state (freshly added, but not
staged / saved yet, these are used in the simulation). The Grok
highlighting is only available in the simulation table when the Grok
processor if in a **draft** state, this is because we can't cleanly
differentiate between staged processors at the moment which one exactly
is being edited (multiple edit panels can be opened at once). I have
discussed this with @tonyghiani and @patpscal and we've agreed this is
fine to start with, with
elastic#218955 and
moving to only one edit at a time we will be able to roll this out to
editing staged processors as well as the draft processor.

- Custom sample data is **always** available, so there is always a way
to test the patterns with highlights etc.

- In the Grok UI package there is a read only version of the sample
component, this is for things like cell rendering within a table and
provides it's tooltips via EUI. There is also an "input" version which
is backed by Monaco (tooltips also provided by Monaco). The styling
differs on the tooltips a bit between the two for this reason, I will
try and enhance / override the Monaco styles so they adhere to our
colour palette / themes etc in a followup.

- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint
to fetch the patterns, however I couldn't find a convenient way to do
this yet without adding an API endpoint somewhere (since this needs an
ES client). I've deferred this for now, but it's why I've preempted
`setup()` being modelled as `async`.

- I had to pass `formControl` around in a bit of an ugly way in the
patterns editor, this was purely due to the `useForm()` hook not working
as I expected in that part of the hierarchy.

- We only need a single `GrokCollection` instance, as such this is
stored in the top level state machine.

- We technically have two areas where state is managed. We have the
state machines and then we have the form state managed by React Hook
Form. The form state is synchronised back to the state machines when
processor changes are made. This Grok integration work introduces a new
need which is for resources to be shared between the processors panel
and the simulation panel, in this case the same `DraftGrokExpression`
instances. There are multiple ways this could have been modelled, with
various pros and cons, but I wanted to keep the paradigm we have now
which is that the processor definitions in the state machine are
"complete" (in the sense they're converted) and not partial form state.
The `DraftGrokExpressions` are integrated directly with the form state,
and synchronised back to a new `resources` property in the state
machine.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
kibanamachine added a commit that referenced this pull request May 29, 2025
…221887)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[Streams] Grok UI integration with enrichment page
(#219146)](#219146)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Kerry
Gallagher","email":"kerry.gallagher@elastic.co"},"sourceCommit":{"committedDate":"2025-05-29T08:07:31Z","message":"[Streams]
Grok UI integration with enrichment page (#219146)\n\n##
Summary\n\nCloses
https://github.com/elastic/streams-program/issues/173.\n\nFrom a high
level this takes the UI components / models in the Grok UI\npackage from
a POC level to a polished level, and then integrates those\nwith the
Enrichment / Extraction page in the Streams UI. The Grok UI\npackage is
completely agnostic to it's consumer, so these package\nresources should
be ready to integrate with any part of Kibana (we want\nto eventually
bring this to Discover, for example).\n\n## Notes for the
reviewer\n\nQuite a few notes, but hopefully they help navigate the code
and\ndecisions made.\n\n- The `PreviewTable` now allows a custom
`renderCellValue` prop, this is\nused when we are dealing with a
**draft** Grok processor.\n\n- With processors we have a saved state
(these don't run in the\nsimulation), a staged state (configured and
added but not saved, these\nare used in the simulation), and a draft
state (freshly added, but not\nstaged / saved yet, these are used in the
simulation). The Grok\nhighlighting is only available in the simulation
table when the Grok\nprocessor if in a **draft** state, this is because
we can't cleanly\ndifferentiate between staged processors at the moment
which one exactly\nis being edited (multiple edit panels can be opened
at once). I have\ndiscussed this with @tonyghiani and @patpscal and
we've agreed this is\nfine to start with,
with\nhttps://github.com//issues/218955?reload=1?reload=1
and\nmoving to only one edit at a time we will be able to roll this out
to\nediting staged processors as well as the draft processor.\n\n-
Custom sample data is **always** available, so there is always a way\nto
test the patterns with highlights etc.\n\n- In the Grok UI package there
is a read only version of the sample\ncomponent, this is for things like
cell rendering within a table and\nprovides it's tooltips via EUI. There
is also an \"input\" version which\nis backed by Monaco (tooltips also
provided by Monaco). The styling\ndiffers on the tooltips a bit between
the two for this reason, I will\ntry and enhance / override the Monaco
styles so they adhere to our\ncolour palette / themes etc in a
followup.\n\n- Giorgos had suggested using the `/_ingest/processor/grok`
ES endpoint\nto fetch the patterns, however I couldn't find a convenient
way to do\nthis yet without adding an API endpoint somewhere (since this
needs an\nES client). I've deferred this for now, but it's why I've
preempted\n`setup()` being modelled as `async`.\n\n- I had to pass
`formControl` around in a bit of an ugly way in the\npatterns editor,
this was purely due to the `useForm()` hook not working\nas I expected
in that part of the hierarchy.\n\n- We only need a single
`GrokCollection` instance, as such this is\nstored in the top level
state machine.\n\n- We technically have two areas where state is
managed. We have the\nstate machines and then we have the form state
managed by React Hook\nForm. The form state is synchronised back to the
state machines when\nprocessor changes are made. This Grok integration
work introduces a new\nneed which is for resources to be shared between
the processors panel\nand the simulation panel, in this case the same
`DraftGrokExpression`\ninstances. There are multiple ways this could
have been modelled, with\nvarious pros and cons, but I wanted to keep
the paradigm we have now\nwhich is that the processor definitions in the
state machine are\n\"complete\" (in the sense they're converted) and not
partial form state.\nThe `DraftGrokExpressions` are integrated directly
with the form state,\nand synchronised back to a new `resources`
property in the state\nmachine.\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Joe
Reuter
<johannes.reuter@elastic.co>","sha":"216ac7e83eb4edb19482b3e0112fe64c4204a938","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","Feature:Streams","v9.1.0","v8.19.0"],"title":"[Streams]
Grok UI integration with enrichment
page","number":219146,"url":"https://github.com/elastic/kibana/pull/219146","mergeCommit":{"message":"[Streams]
Grok UI integration with enrichment page (#219146)\n\n##
Summary\n\nCloses
https://github.com/elastic/streams-program/issues/173.\n\nFrom a high
level this takes the UI components / models in the Grok UI\npackage from
a POC level to a polished level, and then integrates those\nwith the
Enrichment / Extraction page in the Streams UI. The Grok UI\npackage is
completely agnostic to it's consumer, so these package\nresources should
be ready to integrate with any part of Kibana (we want\nto eventually
bring this to Discover, for example).\n\n## Notes for the
reviewer\n\nQuite a few notes, but hopefully they help navigate the code
and\ndecisions made.\n\n- The `PreviewTable` now allows a custom
`renderCellValue` prop, this is\nused when we are dealing with a
**draft** Grok processor.\n\n- With processors we have a saved state
(these don't run in the\nsimulation), a staged state (configured and
added but not saved, these\nare used in the simulation), and a draft
state (freshly added, but not\nstaged / saved yet, these are used in the
simulation). The Grok\nhighlighting is only available in the simulation
table when the Grok\nprocessor if in a **draft** state, this is because
we can't cleanly\ndifferentiate between staged processors at the moment
which one exactly\nis being edited (multiple edit panels can be opened
at once). I have\ndiscussed this with @tonyghiani and @patpscal and
we've agreed this is\nfine to start with,
with\nhttps://github.com//issues/218955?reload=1?reload=1
and\nmoving to only one edit at a time we will be able to roll this out
to\nediting staged processors as well as the draft processor.\n\n-
Custom sample data is **always** available, so there is always a way\nto
test the patterns with highlights etc.\n\n- In the Grok UI package there
is a read only version of the sample\ncomponent, this is for things like
cell rendering within a table and\nprovides it's tooltips via EUI. There
is also an \"input\" version which\nis backed by Monaco (tooltips also
provided by Monaco). The styling\ndiffers on the tooltips a bit between
the two for this reason, I will\ntry and enhance / override the Monaco
styles so they adhere to our\ncolour palette / themes etc in a
followup.\n\n- Giorgos had suggested using the `/_ingest/processor/grok`
ES endpoint\nto fetch the patterns, however I couldn't find a convenient
way to do\nthis yet without adding an API endpoint somewhere (since this
needs an\nES client). I've deferred this for now, but it's why I've
preempted\n`setup()` being modelled as `async`.\n\n- I had to pass
`formControl` around in a bit of an ugly way in the\npatterns editor,
this was purely due to the `useForm()` hook not working\nas I expected
in that part of the hierarchy.\n\n- We only need a single
`GrokCollection` instance, as such this is\nstored in the top level
state machine.\n\n- We technically have two areas where state is
managed. We have the\nstate machines and then we have the form state
managed by React Hook\nForm. The form state is synchronised back to the
state machines when\nprocessor changes are made. This Grok integration
work introduces a new\nneed which is for resources to be shared between
the processors panel\nand the simulation panel, in this case the same
`DraftGrokExpression`\ninstances. There are multiple ways this could
have been modelled, with\nvarious pros and cons, but I wanted to keep
the paradigm we have now\nwhich is that the processor definitions in the
state machine are\n\"complete\" (in the sense they're converted) and not
partial form state.\nThe `DraftGrokExpressions` are integrated directly
with the form state,\nand synchronised back to a new `resources`
property in the state\nmachine.\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Joe
Reuter
<johannes.reuter@elastic.co>","sha":"216ac7e83eb4edb19482b3e0112fe64c4204a938"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219146","number":219146,"mergeCommit":{"message":"[Streams]
Grok UI integration with enrichment page (#219146)\n\n##
Summary\n\nCloses
https://github.com/elastic/streams-program/issues/173.\n\nFrom a high
level this takes the UI components / models in the Grok UI\npackage from
a POC level to a polished level, and then integrates those\nwith the
Enrichment / Extraction page in the Streams UI. The Grok UI\npackage is
completely agnostic to it's consumer, so these package\nresources should
be ready to integrate with any part of Kibana (we want\nto eventually
bring this to Discover, for example).\n\n## Notes for the
reviewer\n\nQuite a few notes, but hopefully they help navigate the code
and\ndecisions made.\n\n- The `PreviewTable` now allows a custom
`renderCellValue` prop, this is\nused when we are dealing with a
**draft** Grok processor.\n\n- With processors we have a saved state
(these don't run in the\nsimulation), a staged state (configured and
added but not saved, these\nare used in the simulation), and a draft
state (freshly added, but not\nstaged / saved yet, these are used in the
simulation). The Grok\nhighlighting is only available in the simulation
table when the Grok\nprocessor if in a **draft** state, this is because
we can't cleanly\ndifferentiate between staged processors at the moment
which one exactly\nis being edited (multiple edit panels can be opened
at once). I have\ndiscussed this with @tonyghiani and @patpscal and
we've agreed this is\nfine to start with,
with\nhttps://github.com//issues/218955?reload=1?reload=1
and\nmoving to only one edit at a time we will be able to roll this out
to\nediting staged processors as well as the draft processor.\n\n-
Custom sample data is **always** available, so there is always a way\nto
test the patterns with highlights etc.\n\n- In the Grok UI package there
is a read only version of the sample\ncomponent, this is for things like
cell rendering within a table and\nprovides it's tooltips via EUI. There
is also an \"input\" version which\nis backed by Monaco (tooltips also
provided by Monaco). The styling\ndiffers on the tooltips a bit between
the two for this reason, I will\ntry and enhance / override the Monaco
styles so they adhere to our\ncolour palette / themes etc in a
followup.\n\n- Giorgos had suggested using the `/_ingest/processor/grok`
ES endpoint\nto fetch the patterns, however I couldn't find a convenient
way to do\nthis yet without adding an API endpoint somewhere (since this
needs an\nES client). I've deferred this for now, but it's why I've
preempted\n`setup()` being modelled as `async`.\n\n- I had to pass
`formControl` around in a bit of an ugly way in the\npatterns editor,
this was purely due to the `useForm()` hook not working\nas I expected
in that part of the hierarchy.\n\n- We only need a single
`GrokCollection` instance, as such this is\nstored in the top level
state machine.\n\n- We technically have two areas where state is
managed. We have the\nstate machines and then we have the form state
managed by React Hook\nForm. The form state is synchronised back to the
state machines when\nprocessor changes are made. This Grok integration
work introduces a new\nneed which is for resources to be shared between
the processors panel\nand the simulation panel, in this case the same
`DraftGrokExpression`\ninstances. There are multiple ways this could
have been modelled, with\nvarious pros and cons, but I wanted to keep
the paradigm we have now\nwhich is that the processor definitions in the
state machine are\n\"complete\" (in the sense they're converted) and not
partial form state.\nThe `DraftGrokExpressions` are integrated directly
with the form state,\nand synchronised back to a new `resources`
property in the state\nmachine.\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Joe
Reuter
<johannes.reuter@elastic.co>","sha":"216ac7e83eb4edb19482b3e0112fe64c4204a938"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: Kerry Gallagher <kerry.gallagher@elastic.co>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com>
zacharyparikh pushed a commit to zacharyparikh/kibana that referenced this pull request Jun 4, 2025
## Summary

Closes elastic/streams-program#173.

From a high level this takes the UI components / models in the Grok UI
package from a POC level to a polished level, and then integrates those
with the Enrichment / Extraction page in the Streams UI. The Grok UI
package is completely agnostic to it's consumer, so these package
resources should be ready to integrate with any part of Kibana (we want
to eventually bring this to Discover, for example).

## Notes for the reviewer

Quite a few notes, but hopefully they help navigate the code and
decisions made.

- The `PreviewTable` now allows a custom `renderCellValue` prop, this is
used when we are dealing with a **draft** Grok processor.

- With processors we have a saved state (these don't run in the
simulation), a staged state (configured and added but not saved, these
are used in the simulation), and a draft state (freshly added, but not
staged / saved yet, these are used in the simulation). The Grok
highlighting is only available in the simulation table when the Grok
processor if in a **draft** state, this is because we can't cleanly
differentiate between staged processors at the moment which one exactly
is being edited (multiple edit panels can be opened at once). I have
discussed this with @tonyghiani and @patpscal and we've agreed this is
fine to start with, with
elastic#218955 and
moving to only one edit at a time we will be able to roll this out to
editing staged processors as well as the draft processor.

- Custom sample data is **always** available, so there is always a way
to test the patterns with highlights etc.

- In the Grok UI package there is a read only version of the sample
component, this is for things like cell rendering within a table and
provides it's tooltips via EUI. There is also an "input" version which
is backed by Monaco (tooltips also provided by Monaco). The styling
differs on the tooltips a bit between the two for this reason, I will
try and enhance / override the Monaco styles so they adhere to our
colour palette / themes etc in a followup.

- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint
to fetch the patterns, however I couldn't find a convenient way to do
this yet without adding an API endpoint somewhere (since this needs an
ES client). I've deferred this for now, but it's why I've preempted
`setup()` being modelled as `async`.

- I had to pass `formControl` around in a bit of an ugly way in the
patterns editor, this was purely due to the `useForm()` hook not working
as I expected in that part of the hierarchy.

- We only need a single `GrokCollection` instance, as such this is
stored in the top level state machine.

- We technically have two areas where state is managed. We have the
state machines and then we have the form state managed by React Hook
Form. The form state is synchronised back to the state machines when
processor changes are made. This Grok integration work introduces a new
need which is for resources to be shared between the processors panel
and the simulation panel, in this case the same `DraftGrokExpression`
instances. There are multiple ways this could have been modelled, with
various pros and cons, but I wanted to keep the paradigm we have now
which is that the processor definitions in the state machine are
"complete" (in the sense they're converted) and not partial form state.
The `DraftGrokExpressions` are integrated directly with the form state,
and synchronised back to a new `resources` property in the state
machine.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
nickpeihl pushed a commit to nickpeihl/kibana that referenced this pull request Jun 12, 2025
## Summary

Closes elastic/streams-program#173.

From a high level this takes the UI components / models in the Grok UI
package from a POC level to a polished level, and then integrates those
with the Enrichment / Extraction page in the Streams UI. The Grok UI
package is completely agnostic to it's consumer, so these package
resources should be ready to integrate with any part of Kibana (we want
to eventually bring this to Discover, for example).

## Notes for the reviewer

Quite a few notes, but hopefully they help navigate the code and
decisions made.

- The `PreviewTable` now allows a custom `renderCellValue` prop, this is
used when we are dealing with a **draft** Grok processor.

- With processors we have a saved state (these don't run in the
simulation), a staged state (configured and added but not saved, these
are used in the simulation), and a draft state (freshly added, but not
staged / saved yet, these are used in the simulation). The Grok
highlighting is only available in the simulation table when the Grok
processor if in a **draft** state, this is because we can't cleanly
differentiate between staged processors at the moment which one exactly
is being edited (multiple edit panels can be opened at once). I have
discussed this with @tonyghiani and @patpscal and we've agreed this is
fine to start with, with
elastic#218955 and
moving to only one edit at a time we will be able to roll this out to
editing staged processors as well as the draft processor.

- Custom sample data is **always** available, so there is always a way
to test the patterns with highlights etc.

- In the Grok UI package there is a read only version of the sample
component, this is for things like cell rendering within a table and
provides it's tooltips via EUI. There is also an "input" version which
is backed by Monaco (tooltips also provided by Monaco). The styling
differs on the tooltips a bit between the two for this reason, I will
try and enhance / override the Monaco styles so they adhere to our
colour palette / themes etc in a followup.

- Giorgos had suggested using the `/_ingest/processor/grok` ES endpoint
to fetch the patterns, however I couldn't find a convenient way to do
this yet without adding an API endpoint somewhere (since this needs an
ES client). I've deferred this for now, but it's why I've preempted
`setup()` being modelled as `async`.

- I had to pass `formControl` around in a bit of an ugly way in the
patterns editor, this was purely due to the `useForm()` hook not working
as I expected in that part of the hierarchy.

- We only need a single `GrokCollection` instance, as such this is
stored in the top level state machine.

- We technically have two areas where state is managed. We have the
state machines and then we have the form state managed by React Hook
Form. The form state is synchronised back to the state machines when
processor changes are made. This Grok integration work introduces a new
need which is for resources to be shared between the processors panel
and the simulation panel, in this case the same `DraftGrokExpression`
instances. There are multiple ways this could have been modelled, with
various pros and cons, but I wanted to keep the paradigm we have now
which is that the processor definitions in the state machine are
"complete" (in the sense they're converted) and not partial form state.
The `DraftGrokExpressions` are integrated directly with the form state,
and synchronised back to a new `resources` property in the state
machine.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:version Backport to applied version labels Feature:Streams This is the label for the Streams Project release_note:skip Skip the PR/issue when compiling release notes v8.19.0 v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants