[Discover] Fix formatting and sorting for custom ES|QL vars

[jughosta]

• Closes [Discover] [ES|QL] Client side sorting doesnt work for fields not in the index #208020

Summary

By default the data view (which is used in ES|QL mode) has only mapped fields as per field caps for the current index pattern. This PR dynamically extends with additional fields based on ES|QL meta information.

This change helps to fix the formatting for ES|QL var values and fixes sorting on them.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[stratoula]

yes this works great now 🙌 Thanx a ton Julia fir working on it (I didnt check the code, just tested the feature!)

[davismcphee]

Code changes look good, and it seems like it will also solve some other subtle issues, thanks! We might want to consider if this could be centralized at some point so ES|QL based data views include custom fields directly in their field lists, but not something for now at least.

[davismcphee]

I partly wonder if this would be more intuitive on the data views service instead since it doesn't actually modify the data view, but I don't feel strongly about it

[jughosta]

@davismcphee Yes, it might be a good location for it too. I wanted to keep it rather as a sync function but getting shortDotsEnable would require to make it async if defined for data views service.

Also should have mentioned it earlier, having this new create method allows to create data view field instances also from packages. DataViewField class is defined in the plugin and can be instantiated directly only in a plugin - we should probably move the whole class to a its own package at some point 🙃

[davismcphee]

I wanted to keep it rather as a sync function but getting shortDotsEnable would require to make it async if defined for data views service.

Good point, I also ran into a similar issue when working on the "All logs" data view since I originally tried to add the cloning logic directly in the data views service as a sync method. There seemed to be a way to work around this by modifying the UI settings abstraction used by the data views service, but I decided against it to avoid expanding the scope of that work.

DataViewField class is defined in the plugin and cannot be instantiated directly only in a plugin - we should probably move the whole class to a its own package at some point 🙃

++ This would probably be the best approach. I didn't clue in that this was a workaround for that limitation, but it would certainly be easier to manage if we could import and instantiate directly from packages.

[jughosta]

@davismcphee Agree, centralizing it would be better. I considered doing it now too but there are some nuances:

• Mutating the current data view fields list might create some unexpected side effects. One of them is what to do with the initial time field vs what is returned in the query response. Should the data view keep its time field or discard it if it's not present in the response. How would it affect the time picker.
• Having a separate data view with only fields based on ES|QL response could be an option too. But creating a data view is an async operation and cannot be simply done during the rendering process. We would need to store it somewhere and most probably pass as an extra prop to UnifiedDataTable and UnifiedDocViewer which is cumbersome.

As a side note here, there is definitely a room for optimizing ES|QL mode by skipping fields fetching when ES|QL data view is created. It would work faster. But we would still need to fetch the time field separately I guess.

[davismcphee]

@jughosta Thanks for the additional context, it's insightful. I'm actually encountering similar obstacles with the state management work related to data view async methods too. I created a separate issue from those observations so we don't lose track of them: #210464. Please feel free to reformat or add any additional context to the issue since you've been working on it lately.

[kibanamachine]

Starting backport for target branches: 8.18, 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13260252055

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 85555eb

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
aiops	606	607	+1
apm	1702	1703	+1
canvas	1226	1227	+1
cloudSecurityPosture	792	793	+1
dashboard	728	729	+1
data	623	624	+1
datasetQuality	282	283	+1
dataViewManagement	223	224	+1
dataVisualizer	854	855	+1
discover	1099	1100	+1
esql	211	212	+1
esqlDataGrid	517	518	+1
eventAnnotationListing	744	745	+1
infra	1303	1304	+1
investigateApp	354	355	+1
lens	1819	1820	+1
lists	431	432	+1
logsShared	362	363	+1
maps	1338	1339	+1
ml	2217	2218	+1
observability	1347	1348	+1
observabilityAIAssistantApp	526	527	+1
presentationUtil	148	149	+1
searchPlayground	295	296	+1
securitySolution	6697	6698	+1
slo	993	994	+1
stackAlerts	273	274	+1
unifiedDocViewer	280	281	+1
unifiedHistogram	286	287	+1
unifiedSearch	395	396	+1
total			+30

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/data-view-utils	6	11	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
cloudSecurityPosture	537.4KB	537.7KB	+257.0B
discover	847.1KB	847.3KB	+169.0B
esqlDataGrid	190.5KB	190.8KB	+256.0B
securitySolution	21.4MB	21.4MB	+1.5KB
slo	877.6KB	877.8KB	+260.0B
unifiedDocViewer	148.2KB	148.5KB	+254.0B
total			+2.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
cloudSecurityPosture	18.9KB	18.8KB	-139.0B
dataViews	62.9KB	62.9KB	+39.0B
esqlDataGrid	9.4KB	9.3KB	-139.0B
total			-239.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/data-view-utils	8	13	+5
dataViews	1229	1233	+4
total			+9

History

• 💚 Build #274747 succeeded 661a7fb
• 💛 Build #273909 was flaky 831b6e9
• 💛 Build #273537 was flaky c41c84d
• 💔 Build #273223 failed 9c5f5d5
• 💔 Build #273210 failed 79a9afe
• 💔 Build #273146 failed b45825a

cc @jughosta

[kibanamachine]

💔 Some backports could not be created

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
❌	8.x	Backport failed because of merge conflicts
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 209360

Questions ?

Please refer to the Backport tool documentation

[jughosta]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation