[Streams] Add client-side field name validation for wired streams

[flash1293]

Summary

Adds client-side validation to the "Add field" flyout in the streams schema editor for wired streams, matching the server-side validation behavior:

• Field names must be namespaced (start with attributes., body.structured., scope.attributes., or resource.attributes.) OR be in the keepFields list
• Field names cannot be OTel reserved fields (body, attributes, scope, resource, span.id, message, trace.id, log.level)

This provides immediate feedback to users when entering invalid field names, rather than waiting for a server error.

Changes

• Export otelReservedFields and isOtelReservedField from @kbn/streams-schema
• Add validation rules to FieldNameSelector in add_field_flyout.tsx
• Add unit tests for validation behavior (14 tests)
• Add Scout UI integration test for field name validation

Test plan

• Open streams schema editor for a wired stream
• Click "Add field"
• Enter a non-namespaced field like foo - should show error about namespaced ECS/OTel schema
• Enter body - should show error about OTel reserved field
• Enter attributes.foo - should be accepted (valid namespace)
• Verify classic streams do not have these restrictions

Closes: https://github.com/elastic/streams-program/issues/472

Made with Cursor

[flash1293]

Self-Review

Overall Assessment

The implementation correctly adds client-side field name validation for wired streams, matching the server-side validation behavior.

What's Good

1. Validation logic correctly mirrors server-side behavior - The order of validation (namespace check first, then OTel reserved check) matches validate_fields.ts on the server.

2. Clean separation of concerns - Added otelReservedFields and isOtelReservedField to @kbn/streams-schema for reuse, keeping the validation helpers centralized.

3. Comprehensive test coverage - 14 unit tests cover various scenarios including:

   • Non-namespaced fields being rejected
   • OTel reserved fields being rejected (with correct error message depending on whether they're in keepFields)
   • All namespace prefixes being accepted
   • keepFields being accepted
   • Classic streams not having these restrictions

4. Good Scout UI test - The integration test validates the full user flow including error display and successful field addition.

Concerns / Areas for Improvement

1. Missing validation for automatic alias conflicts - The server also validates this in validateAncestorFields (lines 47-57):

   for (const prefix of namespacePrefixes) {
     const prefixedName = \`${prefix}${fieldName}\`;
     if (Object.hasOwn(fields, prefixedName) || Object.hasOwn(ancestor.ingest.wired.fields, prefixedName)) {
       throw new MalformedFieldsError(\`Field ${fieldName} is an automatic alias of ${prefixedName}...\`);
     }
   }

   This check isn't implemented in the UI validation. For example, if attributes.foo already exists and the user tries to add foo, the server would reject it but the UI wouldn't show an error. However, this validation requires context about what fields exist in the stream and ancestors - it's not a pure field name validation. The current implementation focuses on the static validation rules. This could be added as a follow-up if needed.

2. otelReservedFields duplication - The otelReservedFields list is manually defined in namespaced_ecs.ts and must be kept in sync with baseMappings in logs_layer.ts. A comment references this but there's no compile-time guarantee. Consider importing or generating from the source of truth in a follow-up.

3. Minor: Test file location - The test file add_field_flyout.test.tsx is placed alongside the component (correct pattern), but there's an existing index.test.tsx in the parent directory that tests the SchemaEditor. The new tests could have potentially been added there for consistency, though a separate focused test file is also reasonable.

Verification

• Unit tests pass locally (14 tests)
• The implementation matches the expected behavior described in the issue
• Classic streams correctly skip the wired-stream-specific validations

Self-review by Cursor

[flash1293]

/ralph Please address the "otelReservedFields duplication" point of the review, I agree that this isn't great

[flash1293]

Follow-up: Address otelReservedFields duplication concern

I've addressed the "otelReservedFields duplication" point from the self-review by adding a sync test.

What was done

1. Added a test in logs_layer.test.ts that verifies Object.keys(baseMappings) matches otelReservedFields:

   • If someone adds/removes a field from baseMappings, the test will fail unless they also update otelReservedFields
   • This provides test-time verification without adding runtime overhead

2. Updated the comment in namespaced_ecs.ts to reference the sync test, making it clear to future developers that:

   • The list must match baseMappings keys
   • There's a test that enforces this

Why this approach

• @kbn/streams-schema is a shared-common package usable in both browser and server
• logs_layer.ts is server-only and imports from @kbn/streams-schema
• Cannot have @kbn/streams-schema import from the plugin (would create circular dependency)
• Keeping the list in @kbn/streams-schema allows UI code to use it for validation
• The test ensures sync at test-time without runtime cost

Alternative considered

Could have extracted the reserved fields list directly from baseMappings at module load time, but:

• Would require moving more code to the shared package
• Current approach is simpler and achieves the same compile-time safety guarantee through tests

Tests verified

• All 6 tests in logs_layer.test.ts pass (including the new sync test)
• All 14 tests in add_field_flyout.test.tsx pass
• ESLint passes on changed files

Self-review by Cursor

[flash1293]

/ralph The "Add field" button in the flyout should be disabled if the validation fails.

[flash1293]

Follow-up: Disable Add field button when validation fails

Addressed the request to disable the "Add field" button when validation fails.

Changes

add_field_flyout.tsx

• Changed the button's isDisabled prop from methods.formState.isSubmitted && !methods.formState.isValid to just !methods.formState.isValid
• This provides immediate feedback to users - the button is now disabled whenever the form validation fails (invalid field name or missing type), rather than only after a failed submission attempt

add_field_flyout.test.tsx

• Added 3 new tests for button disabled state behavior:
  • Button disabled when field name validation fails
  • Button disabled when field name is valid but type is not selected
  • Button enabled when all required fields are valid

Testing

• All 17 tests pass
• ESLint passes

[elasticmachine]

Pinging @elastic/streams-program-team (Team:streams-program)

[mohamedhamed-ahmed]

LGTM! Thank you

[flash1293]

/ci-ralph

[flash1293]

/ralph check the test failures of build 398525 and fix them

Build URL: https://buildkite.com/elastic/kibana-pull-request/builds/398525

Use ci/bk-build-info.sh 398525 to fetch detailed build failure information.

[flash1293]

Test fix for Scout UI failure

Fixed the failing Scout test validates field names for wired streams in wired_streams_schema.spec.ts.

Root cause

After the change to disable the "Add field" button when form validation fails, the test at line 217 was trying to click a disabled button to verify the flyout stays open. Playwright blocks clicks on disabled elements by default, causing a timeout.

Fix

Updated the test to directly assert that the button is disabled (await expect(addButton).toBeDisabled()) instead of clicking it and checking if the flyout remains visible. This is actually a better test because it directly verifies the expected behavior (button disabled) rather than inferring it (flyout stays open).

The rest of the test flow remains unchanged - it still verifies that after entering a valid field name and type, the button becomes enabled and clicking it successfully adds the field.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: c0152e4

Failed CI Steps

• Scout: [ observability / observability ] plugin

Test Failures

• [job] [logs] Scout: [ observability / observability ] plugin / local-serverless-observability_complete - Rule Details Page - Admin - should display dashboard options in related dashboards dropdown when editing rule

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/streams-schema	284	285	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
streamsApp	1.7MB	1.7MB	+761.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/streams-schema	302	305	+3

History

• 💔 Build #398525 failed 973e5a2