[One Workflow] Add `entries` Liquid filter for iterating over object keys by shahargl · Pull Request #259249 · elastic/kibana

shahargl · 2026-03-24T07:20:05Z

Summary

Adds a custom entries Liquid filter to the workflow templating engine that converts an object/map into an array of {key, value} pairs. This enables foreach to iterate over object keys — a common need when working with Elasticsearch API responses that return object maps (e.g. _settings, _aliases, _mappings).

Before (multi-step workaround):

# 20+ lines of Liquid string building + data.set + json_parse
- name: reshape
  type: data.set
  with:
    entries: >-
      {%- liquid
        assign result = ""
        for pair in steps.get_settings.output.body
          ...
        endfor
        echo "[" | append: result | append: "]"
      -%}
- name: parse
  type: data.set
  with:
    parsed: "${{ variables.entries | json_parse }}"
- name: iterate
  type: foreach
  foreach: "{{ variables.parsed }}"

After (one-liner):

- name: iterate
  type: foreach
  foreach: "{{ steps.get_settings.output.body | entries }}"
  steps:
    - name: process
      type: console
      with:
        message: "Index: {{ foreach.item.key }}, ILM: {{ foreach.item.value.settings.index.lifecycle.name }}"

Changes

Core filter (server + client):

Register entries filter in WorkflowTemplatingEngine (server-side execution)
Register entries filter in client-side validation (validate_liquid_template.ts)
Register entries filter in client-side hover expression evaluator (evaluate_expression.ts)
Add entries to Monaco autocomplete suggestions (liquid_completions.ts)

Foreach UI improvements (bonus):

Fix foreach schema resolver to recognize | entries on objects and produce {key, value} item schema instead of showing "Expected array for foreach iteration, but got object"
Fix foreach.item hover resolution — the server only persists {index, total} in foreach state (not items). The client now re-evaluates the foreach expression from input.foreach to derive items, matching the server-side buildForeachContext behavior
Lazy-load foreach step input when hovering over foreach.* paths, since step I/O is lazy-loaded and wasn't being fetched for non-steps.* paths
Increase hover tooltip max depth from 5 to 8 to prevent deeply nested values (common with entries output) from being truncated to "{Object with N properties}"

Documentation:

Update filter_contribution.md with two missing steps: client-side hover evaluator registration and foreach schema awareness for type-changing filters

Testing

8 new unit tests for entries filter in templating_engine.test.ts (objects, nested values, arrays passthrough, null, empty objects, evaluateExpression, filter chaining)
2 new tests for entries in get_foreach_state_schema.test.ts (simple object + deeply nested object with spaces)

References

Closes elastic/security-team#16426

Made with Cursor

Registers a custom `entries` filter in the workflow templating engine that converts an object/map into an array of {key, value} pairs. This enables `foreach` to iterate over object keys — a common need when working with Elasticsearch API responses that return object maps (e.g. _settings, _aliases). Closes elastic/security-team#16426 Made-with: Cursor

- Register `entries` filter in client-side Liquid engine so hover evaluations don't fail on unknown filter - Teach the foreach schema resolver to recognize `| entries` on objects and produce {key, value} item schema instead of throwing - Add test for entries filter in foreach schema resolution Made-with: Cursor

The server-side execution engine only persists {index, total} in foreach step state — not the full items array. The client-side hover provider was looking for state.items which never exists, causing foreach.item to always show as "undefined in the current execution context". Fix: extract the foreach expression from the step's input (stored as input.foreach) and re-evaluate it against the execution context using LiquidJS — matching what the server-side context manager does in buildForeachContext/resolveForeachItems. Made-with: Cursor

The entries filter wraps values in {key, value} objects adding 2 extra nesting levels. With maxDepth=5, deeply nested ES responses (e.g. settings.index.lifecycle.name) get truncated to "{Object with N properties}" in the hover tooltip. Bump to 8 to accommodate. Made-with: Cursor

When hovering over foreach.* paths, the foreach step's input data (containing the foreach expression) may not be loaded yet since step I/O is lazy-loaded on hover. Trigger input loading for foreach steps before evaluating the expression, so findForeachContext can re-evaluate the foreach expression and resolve foreach.item correctly. Made-with: Cursor

The guide was missing two critical steps that caused real issues: - Step 3: Client-side hover expression evaluator registration - Step 5: Foreach schema awareness for type-changing filters Without these, new filters cause broken hover tooltips and misleading "Expected array" warnings in the foreach editor. Made-with: Cursor

Copilot

Pull request overview

Adds an entries Liquid filter and wires it through server execution and client editor tooling so foreach can iterate object/map keys (especially useful for Elasticsearch responses).

Changes:

Add entries filter registration on server and in client-side Liquid evaluation/validation + autocomplete.
Improve foreach hover/schema behavior by re-evaluating input.foreach to derive items and lazy-loading foreach step input for hover.
Increase hover truncation depth to better display nested entries output.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/platform/plugins/shared/workflows_management/public/widgets/workflow_yaml_editor/lib/template_expression/resolve_path_value.ts	Increases hover/value truncation depth for better nested display.
src/platform/plugins/shared/workflows_management/public/widgets/workflow_yaml_editor/lib/template_expression/evaluate_expression.ts	Registers `entries` filter client-side and derives foreach items by re-evaluating `input.foreach`.
src/platform/plugins/shared/workflows_management/public/widgets/workflow_yaml_editor/lib/monaco_providers/unified_hover_provider.ts	Lazy-loads foreach step input so hover evaluation can re-derive foreach items.
src/platform/plugins/shared/workflows_management/public/widgets/workflow_yaml_editor/lib/autocomplete/suggestions/liquid/liquid_completions.ts	Adds `entries` to Liquid filter completions.
src/platform/plugins/shared/workflows_management/public/features/workflow_context/lib/get_foreach_state_schema.ts	Updates foreach schema resolver to understand `
src/platform/plugins/shared/workflows_management/public/features/workflow_context/lib/get_foreach_state_schema.test.ts	Adds tests for foreach schema behavior with `
src/platform/plugins/shared/workflows_management/common/lib/validate_liquid_template.ts	Adds `entries` as a no-op filter for client-side validation parsing.
src/platform/plugins/shared/workflows_execution_engine/server/templating_engine.ts	Registers server-side `entries` filter implementation.
src/platform/plugins/shared/workflows_execution_engine/server/templating_engine.test.ts	Adds unit tests for `entries` behavior and chaining.
src/platform/plugins/shared/workflows_execution_engine/server/filter_contribution.md	Documents additional required registration points (hover evaluator + foreach schema).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-24T08:39:06Z

  return extractForeachItemSchemaFromArray(foreachParamParsed);
 };

+const ENTRIES_FILTER = 'entries';


entries is intended for object/maps, but this schema branch only triggers for z.ZodObject. In practice, many map-shaped payloads (e.g., Elasticsearch index-name → settings) are modeled as z.ZodRecord, so foreach: '{{ someMap | entries }}' may still throw 'Expected array...' in the editor. Consider extending the condition to also handle z.ZodRecord (and ideally use the record's value schema for value when available) so the schema matches real-world map usage.

Copilot · 2026-03-24T08:39:06Z

 ): z.ZodType {
  const parsedPath = parseVariablePath(foreachParam);
  const iterateOverPath = parsedPath?.propertyPath;
+  const hasEntriesFilter = parsedPath?.filters?.includes(ENTRIES_FILTER) ?? false;


entries is intended for object/maps, but this schema branch only triggers for z.ZodObject. In practice, many map-shaped payloads (e.g., Elasticsearch index-name → settings) are modeled as z.ZodRecord, so foreach: '{{ someMap | entries }}' may still throw 'Expected array...' in the editor. Consider extending the condition to also handle z.ZodRecord (and ideally use the record's value schema for value when available) so the schema matches real-world map usage.

Copilot · 2026-03-24T08:39:06Z

+    } else if (iterableSchema instanceof z.ZodObject && hasEntriesFilter) {
+      return z.object({ key: z.string(), value: z.unknown() });


entries is intended for object/maps, but this schema branch only triggers for z.ZodObject. In practice, many map-shaped payloads (e.g., Elasticsearch index-name → settings) are modeled as z.ZodRecord, so foreach: '{{ someMap | entries }}' may still throw 'Expected array...' in the editor. Consider extending the condition to also handle z.ZodRecord (and ideally use the record's value schema for value when available) so the schema matches real-world map usage.

Suggested change

} else if (iterableSchema instanceof z.ZodObject && hasEntriesFilter) {

return z.object({ key: z.string(), value: z.unknown() });

} else if (

hasEntriesFilter &&

(iterableSchema instanceof z.ZodObject || iterableSchema instanceof z.ZodRecord)

) {

// When using the "entries" filter on an object or record, we expose { key, value } items.

// For ZodRecord, try to reuse the record's value schema if available; otherwise fall back to unknown.

let valueSchema: z.ZodType = z.unknown();

if (iterableSchema instanceof z.ZodRecord) {

const recordAny = iterableSchema as any;

const recordValueSchema: z.ZodType | undefined =

recordAny.valueType ?? recordAny._def?.valueType;

if (recordValueSchema) {

valueSchema = recordValueSchema;

}

}

return z.object({ key: z.string(), value: valueSchema });

Not reproducible today. I investigated: (1) inferZodType() always produces z.ZodObject for objects, never ZodRecord. (2) elasticsearch.request has no output schema — returns z.unknown(). (3) Typed ES connectors use ZodObject/ZodUnion, not ZodRecord. No current code path produces a ZodRecord here. Will add when we implement output schemas for ES request steps.

Copilot · 2026-03-24T08:39:07Z

+    if (stepData.state && typeof stepData.state.index === 'number') {
+      const index = stepData.state.index as number;
+      const total = (stepData.state.total as number) ?? 0;


When items are re-derived by evaluating input.foreach, total is still taken from stepData.state.total (defaulting to 0 if missing/not a number). This can make foreach.total incorrect in hover/evaluation even though items is available. Consider computing total as the state value only when it's a valid number, otherwise fall back to items.length (both in the legacy state.items branch and the re-evaluation branch).

Valid point. In practice state.total is always set by the server (it's written in enterForeach alongside index), so it should match. But using items.length as a fallback when we've re-derived items is safer. Will address in a follow-up.

Copilot · 2026-03-24T08:39:07Z

+      const foreachExpression = extractForeachExpression(stepData.input);
+      if (foreachExpression) {
+        const items = resolveForeachItems(foreachExpression, context);
+        if (items) {
+          return {
+            item: items[index] ?? (items.length > 0 ? items[0] : null),
+            index,
+            total,
+            items,
+          };
+        }
+      }


When items are re-derived by evaluating input.foreach, total is still taken from stepData.state.total (defaulting to 0 if missing/not a number). This can make foreach.total incorrect in hover/evaluation even though items is available. Consider computing total as the state value only when it's a valid number, otherwise fall back to items.length (both in the legacy state.items branch and the re-evaluation branch).

Copilot · 2026-03-24T08:39:07Z

+    for (const [sId, stepData] of Object.entries(context.steps)) {
+      if (
+        stepData.state &&
+        typeof stepData.state.index === 'number' &&
+        stepData.input === undefined
+      ) {
+        const fullData = await this.fetchStepDataIfNeeded(stepData, sId);
+        if (fullData && fullData !== stepData) {
+          if (!changed) {
+            enrichedSteps = { ...context.steps };
+            changed = true;
+          }
+          enrichedSteps[sId] = fullData;
+        }
+      }
+    }


This loop awaits fetchStepDataIfNeeded sequentially per foreach step. If there are multiple foreach steps (or slow I/O), hover latency can add up. Consider collecting the step IDs that need enrichment and fetching them in parallel (e.g., Promise.all) and then applying updates in one pass.

Suggested change

for (const [sId, stepData] of Object.entries(context.steps)) {

if (

stepData.state &&

typeof stepData.state.index === 'number' &&

stepData.input === undefined

) {

const fullData = await this.fetchStepDataIfNeeded(stepData, sId);

if (fullData && fullData !== stepData) {

if (!changed) {

enrichedSteps = { ...context.steps };

changed = true;

}

enrichedSteps[sId] = fullData;

}

}

}

const entriesToEnrich = Object.entries(context.steps).filter(([_, stepData]) => {

return (

stepData.state &&

typeof stepData.state.index === 'number' &&

stepData.input === undefined

);

});

if (entriesToEnrich.length === 0) {

return context;

}

const enrichedResults = await Promise.all(

entriesToEnrich.map(([sId, stepData]) => this.fetchStepDataIfNeeded(stepData, sId))

);

entriesToEnrich.forEach(([sId, stepData], index) => {

const fullData = enrichedResults[index];

if (fullData && fullData !== stepData) {

if (!changed) {

enrichedSteps = { ...context.steps };

changed = true;

}

enrichedSteps[sId] = fullData;

}

});

In practice there's almost always exactly 1 foreach step (nested foreach is rare), so the sequential await has no meaningful latency impact. The React Query cache also deduplicates — second hover on the same step is a cache hit with no HTTP request. Will parallelize if we see real latency issues.

elasticmachine · 2026-03-24T08:45:49Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: fdeda10

Failed CI Steps

Test Failures

[job] [logs] affected Scout: [ platform / fleet ] plugin / local-stateful-classic - When the user has All privileges for Integrations but None for Fleet - Integrations are visible but cannot be added
[job] [logs] FTR Configs #144 / Search Playground - hosted Unsaved Playground setup chat experience without any indices create index from UI "after all" hook for "add data source and creating an LLM should show chat start page"
[job] [logs] FTR Configs #144 / Search Playground - hosted Unsaved Playground setup chat experience without any indices create index from UI should be able to create index from UI

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`workflowsManagement`	2.0MB	2.0MB	+1.6KB

A foreach step produces multiple step executions with the same stepId (enter, advance per iteration, exit). buildExecutionContext was letting the last one overwrite earlier ones, which could lose the state.index needed by findForeachContext to resolve foreach.item during hover. Now we preserve the execution entry that carries state.index and only merge output/error/status from later executions. Made-with: Cursor

The foreach expression (e.g. "{{ steps.X.output | entries }}") needs both the foreach step's input AND the referenced step's output to evaluate. enrichForeachStepInput was only loading the foreach step's input but not the output of steps it references, causing the expression to evaluate to undefined. Now load I/O for all steps missing output when in a foreach hover path. Made-with: Cursor

shahargl · 2026-03-24T09:28:51Z

Update: Additional commits after initial review

Added 3 more commits addressing issues found during manual testing:

Fixes for `foreach.item` hover resolution (pre-existing bugs)

These are pre-existing bugs unrelated to the entries filter, exposed during testing. Root cause: #253547 (lazy-load step I/O) introduced lazy-loading of step input/output but didn't account for foreach.* hover paths:

Preserve foreach step state in execution context (build_execution_context.ts) — A foreach step produces multiple step execution records with the same stepId (enter, advance, exit). buildExecutionContext was letting the exit execution overwrite the entry, losing state.index needed for foreach.item resolution.
Load referenced step outputs for foreach hover (unified_hover_provider.ts) — The foreach expression (e.g. {{ steps.X.output | entries }}) needs both the foreach step's input AND the referenced step's output. The enrichment was only loading the foreach step's input but not the output of steps it references.

Copilot review responses

ZodRecord support: Not reproducible today — inferZodType always produces ZodObject for consts, and elasticsearch.request has no output schema (returns z.unknown()). Will add when we have connectors that return ZodRecord output schemas.
total fallback to items.length: Valid nit, will address in follow-up.
Parallel Promise.all: Overkill for now — almost always 1 foreach step in practice.

The top-level checklist was missing Tests as a numbered step. Made-with: Cursor

rosomri

lgtm

…keys (elastic#259249) ## Summary Adds a custom `entries` Liquid filter to the workflow templating engine that converts an object/map into an array of `{key, value}` pairs. This enables `foreach` to iterate over object keys — a common need when working with Elasticsearch API responses that return object maps (e.g. `_settings`, `_aliases`, `_mappings`). **Before** (multi-step workaround): ```yaml # 20+ lines of Liquid string building + data.set + json_parse - name: reshape type: data.set with: entries: >- {%- liquid assign result = "" for pair in steps.get_settings.output.body ... endfor echo "[" | append: result | append: "]" -%} - name: parse type: data.set with: parsed: "${{ variables.entries | json_parse }}" - name: iterate type: foreach foreach: "{{ variables.parsed }}" ``` **After** (one-liner): ```yaml - name: iterate type: foreach foreach: "{{ steps.get_settings.output.body | entries }}" steps: - name: process type: console with: message: "Index: {{ foreach.item.key }}, ILM: {{ foreach.item.value.settings.index.lifecycle.name }}" ``` ### Changes **Core filter (server + client):** - Register `entries` filter in `WorkflowTemplatingEngine` (server-side execution) - Register `entries` filter in client-side validation (`validate_liquid_template.ts`) - Register `entries` filter in client-side hover expression evaluator (`evaluate_expression.ts`) - Add `entries` to Monaco autocomplete suggestions (`liquid_completions.ts`) **Foreach UI improvements (bonus):** - Fix foreach schema resolver to recognize `| entries` on objects and produce `{key, value}` item schema instead of showing "Expected array for foreach iteration, but got object" - Fix `foreach.item` hover resolution — the server only persists `{index, total}` in foreach state (not items). The client now re-evaluates the foreach expression from `input.foreach` to derive items, matching the server-side `buildForeachContext` behavior - Lazy-load foreach step input when hovering over `foreach.*` paths, since step I/O is lazy-loaded and wasn't being fetched for non-`steps.*` paths - Increase hover tooltip max depth from 5 to 8 to prevent deeply nested values (common with `entries` output) from being truncated to `"{Object with N properties}"` **Documentation:** - Update `filter_contribution.md` with two missing steps: client-side hover evaluator registration and foreach schema awareness for type-changing filters ### Testing - 8 new unit tests for `entries` filter in `templating_engine.test.ts` (objects, nested values, arrays passthrough, null, empty objects, evaluateExpression, filter chaining) - 2 new tests for `entries` in `get_foreach_state_schema.test.ts` (simple object + deeply nested object with spaces) ## References Closes elastic/security-team#16426 Made with [Cursor](https://cursor.com)

shahargl added 6 commits March 23, 2026 17:19

shahargl requested a review from a team as a code owner March 24, 2026 07:20

shahargl added release_note:enhancement Team:One Workflow Team label for One Workflow (Workflow automation) v9.4.0 backport:skip This PR does not require backporting labels Mar 24, 2026

skynetigor requested a review from Copilot March 24, 2026 08:37

Copilot AI reviewed Mar 24, 2026

View reviewed changes

shahargl added 2 commits March 24, 2026 11:09

rosomri reviewed Mar 24, 2026

View reviewed changes

Comment thread src/platform/plugins/shared/workflows_execution_engine/server/filter_contribution.md Outdated

Comment thread ...shared/workflows_management/public/features/workflow_context/lib/get_foreach_state_schema.ts

Add missing Tests step to filter contribution guide checklist

2ae5380

The top-level checklist was missing Tests as a numbered step. Made-with: Cursor

shahargl requested a review from rosomri March 24, 2026 11:14

rosomri approved these changes Mar 24, 2026

View reviewed changes

shahargl merged commit 62a71dc into elastic:main Mar 24, 2026
18 checks passed

		} else if (iterableSchema instanceof z.ZodObject && hasEntriesFilter) {
		return z.object({ key: z.string(), value: z.unknown() });

-    } else if (iterableSchema instanceof z.ZodObject && hasEntriesFilter) {
-      return z.object({ key: z.string(), value: z.unknown() });
+    } else if (
+      hasEntriesFilter &&
+      (iterableSchema instanceof z.ZodObject || iterableSchema instanceof z.ZodRecord)
+    ) {
+      // When using the "entries" filter on an object or record, we expose { key, value } items.
+      // For ZodRecord, try to reuse the record's value schema if available; otherwise fall back to unknown.
+      let valueSchema: z.ZodType = z.unknown();
+      if (iterableSchema instanceof z.ZodRecord) {
+        const recordAny = iterableSchema as any;
+        const recordValueSchema: z.ZodType | undefined =
+          recordAny.valueType ?? recordAny._def?.valueType;
+        if (recordValueSchema) {
+          valueSchema = recordValueSchema;
+        }
+      }
+      return z.object({ key: z.string(), value: valueSchema });

Conversation

shahargl commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

References

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

shahargl Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

shahargl Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

shahargl Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 24, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

Uh oh!

shahargl commented Mar 24, 2026

Update: Additional commits after initial review

Fixes for foreach.item hover resolution (pre-existing bugs)

Copilot review responses

Uh oh!

Uh oh!

Uh oh!

rosomri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shahargl commented Mar 24, 2026 •

edited

Loading

Fixes for `foreach.item` hover resolution (pre-existing bugs)