[Streams] Schema Editor advanced field mapping options

[Kerry350]

Summary

Closes https://github.com/elastic/streams-program/issues/88

Adds JSON advanced field mapping parameters to the Schema Editor.

Main questions here are around the types and data structure. In this PR these are added as an additionalProperties property, but we may also want to have all of these parameters top level (like type and format). This version makes separating concerns easier in the UI and separating "first class" options vs advanced options, I could see pros and cons to both, and also things might be "upgraded" from advanced to first class later on. Also an open question on whether the MappingProperty type needs to be explicitly redefined for Zod (ES will obviously reject anything that isn't supported here).

[tonyghiani]

Overall looks good, I left some minor notes on the code.

Regarding the additionalProperties vs top-level props, I like the separation of concerns as well, but this introduce potentially a de-sync between the stream and the component template.

For instance, a type defined in the additional properties (copy/paste, human error...) would override the defined type from the selector on the component template, and the user might never notice since the accordion is closed and the stored type in the stream remain the selector one.

As per following screenshot, the stream field would be:
{
    "type": "ip",
    "additionalProperties": {
        "type": "keyword"
    }
}

while in the component template it will be
{
  "host.ip": {
    "type": "keyword"
  }
}

It might be worth excluding the type from the additionalProperties, wdyt?

A last note: when the additional properties have an unacceptable property, the error report is not very helpful, although we get the ES error message in the response, we can we pass it correctly into the error toast for a better feedback.

[Kerry350]

@tonyghiani

Regarding the additionalProperties vs top-level props, I like the separation of concerns as well, but this introduce potentially a de-sync between the stream and the component template.
For instance, a type defined in the additional properties (copy/paste, human error...) would override the defined type from the selector on the component template, and the user might never notice since the accordion is closed and the stored type in the stream remain the selector one.

Yeah, I'd considered this and ultimately thought if someone had explicitly defined something in advanced settings, then it would be okay to override things, but I guess it does cause a divergence in the source of truth for both type and format. I think what I'll do is move them all top level, and then add some functions to the schema package to "get advanced parameters" which will strip our first class properties.

A last note: when the additional properties have an unacceptable property, the error report is not very helpful, although we get the ES error message in the response, we can we pass it correctly into the error toast for a better feedback.

Yeah, that's fair. I think I went down this rabbit hole once before and it was to do with the way streamsRepositoryClient propagates it's error and the information passed on. I'll take a look again.

[tonyghiani]

Yeah, I'd considered this and ultimately thought if someone had explicitly defined something in advanced settings, then it would be okay to override things, but I guess it does cause a divergence in the source of truth for both type and format. I think what I'll do is move them all top level, and then add some functions to the schema package to "get advanced parameters" which will strip our first class properties.

Sounds like a good approach to me 👌

Yeah, that's fair. I think I went down this rabbit hole once before and it was to do with the way streamsRepositoryClient propagates it's error and the information passed on. I'll take a look again.

It might help in this case too, I used something like this to get the error correctly from an ES propagated one:

} catch (error) {
      toasts.addError(new Error(error.body.message), {
        title: i18n.translate(
          'xpack.streams.streamDetailView.managementTab.enrichment.saveChangesError',
          { defaultMessage: "An issue occurred saving processors' changes." }
        ),
        toastMessage: error.body.message,
      });
    }

[Kerry350]

@tonyghiani Ready for another look. I've made all of the parameter handling top level, and then pushed the stripping to getAdvancedParameters.

[tonyghiani]

LGTM, thanks for the changes!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0cb6bbe

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
streamsApp	313	315	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
streamsApp	304.0KB	305.8KB	+1.8KB

History

• 💚 Build #275544 succeeded 9ace472
• 💔 Build #275515 failed fa7757a

cc @Kerry350

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13330592124

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation