[Discover] Supports SQL query language

[stratoula]

Summary

Note: This PR will be merged on a feature branch

This PR enables the SQL queries in Discover. The feature is available after an advanced setting is enabled.

The user can run sql queries and also save them as saved searches. These searches cant be used for creating an aggregation based visualization but can be imported on a dashboard.

For the technical preview we are going to support the following features:

• SQL integration, FROM clause bound to a data view (error if no match)
• No autocomplete
• No date histogram
• Saving as part of the saved search
• No switch to Lens
• No filter by cell click
• Time range integration
• Dashboard filter integration
• No async search / search sessions
• No “add field” button
• No document viewer / context viewer
• Inline documentation

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

[MichaelMarcialis]

Amazing work as usual, @stratoula! I spent some time with it yesterday and have put together an initial list of comments for your review. Let me know if you have any questions or need any design support to achieve any of the following:

Data view and text-based query language menu

• On the "SQL" option in this menu, the mouse cursor changes from pointer to default when hovering the accompanying "Technical preview" badge. Can we fix this so that the mouse cursor remains a constant pointer when hovering the "SQL" option (including the badge portion)?

• Both the "Add a field to this data view" and "Manage this data view" buttons should not be present in the menu when a text-based query language is currently selected.

• In my wireframes, I had suggested that when a text-based query language was selected, we should mark each individual data view option with a warning icon and tooltip. On second thought, after playing with it here, that suggestion seems heavy-handed and creates a lot of potential redundancy. Would it be possible to instead move these individual warning icons and tooltips to a single warning icon and tooltip that prepends the "Data views" title? Doing so will cut down on the redundancy, but still provide the warning to those looking to switch from SQL mode to data view mode.

• After selecting "SQL" and closing the menu, reopening the menu appears to apply focus styles to both the "SQL" option and the and the first data view simultaneously. The browser is only actually focusing the first data view, despite the styling being applied to "SQL" as well. I believe we should only be applying the focus styles to a single option at a time in this menu. Additionally, I imagine in the above example, the "SQL" is the option that should be focused by default when the menu is reopened.

• Would it be possible to adjust the horizontal padding of selectable options in this menu so that they appear flush with the left/right edges of menu? I believe EUI has a popover example with this flush look.

Confirmation modal

• When in SQL mode and attempting to switch back to data view mode, users are presented with a confirmation modal. Interacting with the close/cancel button currently closes the modal and acts as if the user chose to proceed to switch modes without saving (losing their query in the process). This is not what I would expect as a user. Can we instead change this so that interacting with the the close/cancel (X) button simply cancels the mode switch action and keeps the user in SQL mode with their query?

• The "Don't show this warning again" checkbox isn't vertically centered with the opposing buttons that appear to its right side. Would it be possible to center align it vertically?

• The "Switch without saving" and "Save and switch" buttons are not right aligned in the modal currently. Possible to do so?

• The gutter space between the "Switch without saving" and "Save and switch" buttons can be reduced to 16px.

• The "Switch without saving" button appears to be using custom styles. Could we simply use the standard EuiButton with color="warning" prop instead?

• When choosing to "Save and switch", a second modal for saving the search appears on top of the confirmation modal. Could we replace the confirmation modal with the save modal, rather than stacking them? At that point, completing the save modal should then perform the switch. Canceling out of the save modal should be considered the user wanting to cancel and cancel the entire mode switch action.

Editor

General

• Is there an alternative code highlighting theme that we can use which doesn't so prominently utilize blue colored text? I'm worried the current blue color is very similar to the color for links that we use across Kibana.

• Would it be possible to add tooltips for enable/disable word wrap, expand/compact query editor, and SQL reference? Examples can be seen in the wireframes.

• Clicking the SQL reference button once opens it. Clicking it again should close it, but the popover remains open here, requiring users to click elsewhere on the page to close. Would it be possible to change it so that clicking the button when the popover is open will close it?

• The red underlining of errors within the editor appears odd currently. For example, when supplying an unavailable data view with the SQL FROM clause, only the R in FROM will be underlined in red. It seems odd that only a single character in the clause is being underlined. Further, would it be better to have the unavailable data view underlined here instead?

• Can we reduce the width of the line number column in the current error list popover? It feels overly wide currently.

• I had suggested in the wireframes that clicking an individual error in the error list popover should focus the user's cursor on that error in the editor. That doesn't appear to be implemented here. Would it be possible to add?

Compact/auto-sizing editor

• Rather than using primary colored EuiButtonIcon for the expand and SQL reference buttons, can we instead style them to appear visually similar other EUI append and prepend buttons, with the light gray background and blue colored icon (similar to the calendar icon prepended to the date picker)?

Unfocused

• Can we apply the standard rounded corners to the top-left and bottom-left corners of the editor to better match other EUI inputs?

• Would it be possible to reduce the left and right padding of the unfocused editor to 12px?

• The single visible line of query text is not vertically centered in its container currently. Would it be possible to vertically center this text?

• When more than one line of SQL is entered into the editor and the editor is then unfocused, the original design expectation was that the query would be minified down into a single line (i.e. line breaks turned into spaces), with as much of the multi-line query being shown as possible until reaching the rightmost edge of the available space. Only at that point should the query text-overflow to ellipsis. However, what appears to be happening in the PR is that only the first line with appended ellipsis is shown when the editor is unfocused, regardless of length. Can we change to the original intended behavior?

• The bottom border that appears below the editor to indicate either focus (blue) or invalid (red) states appears to be 1px here. Could we change these to be 2px to match the size of similar states in other EUI inputs?

Focused

• Can we apply the standard rounded corners to the top-left, bottom-left, and bottom-right corners of the editor and footer to better match other EUI inputs?

• Would it be possible to apply some vertical padding at the beginning and end of the focused editor? I'm thinking 8px will be sufficient.

• Can we add a light gray background behind the line numbers on the left of the editor for added separation between them and the query?

• When the editor is focused, a scrollbar track for the editor appears to the right, even if the maximum height for the editor has yet to be exceeded. Could we change this behavior to hide the scrollbar track until the user has the need/ability to scroll the editor?

• It looks like the footer container is positioned slightly incorrectly, appearing a few pixels too far to the right. Can this be corrected to align with the editor above?

• Can we add additional padding to the footer area (that contains line count, error count, and run query shortcut)? I'm thinking a total of 8px vertically and 16px horizontally should look nice.

• Can we change the font size for text within the footer (with line count, error count, and run query shortcut) be 12px instead of the current 14px?

• Let's change the "⌘ + Enter" shortcut from being in an EuiBadge to EuiCode (to better match how we display keyboard shortcuts elsewhere in Kibana).

Expanded/manual sizing editor

• Can we apply the standard rounded corners to the top-left, bottom-left, top-right, and bottom-right corners of the editor's header and footer to better match other EUI inputs?

• Can we add additional padding to both the header (word wrap, compact, SQL reference) and footer (line count, error count, and run query shortcut) areas? I'm thinking a total of 8px vertically and 16px horizontally should look nice for both.

• Both the the header (word wrap, compact, SQL reference) and footer (line count, error count, and run query shortcut) areas appear misaligned on the right side. Can this be fixed?

• The word wrap button is currently being given the fill treatment when disabled. Can we remove that fill styling? I believe the icon change and addition of a tooltip should be sufficient in indicating what state the word wrap setting is in.

• The bottom border that typically appears below EUI inputs denoting focus and invalid states is missing when the editor is expanded. Can we add those in?

• It appears that a custom resizer is being used when the editor is in expanded mode. Would it be possible to switch to using EUI's resizable container components to ensure consistent styling?

Additional notes

• When SQL errors are present and the query is run, the current error messaging doesn't currently appear as suggested in the wireframes. The field list still shows (even in situations where a non-existing data view is being referenced) and the only option provided to users is to show the error message in a toast. Would it be possible to change this to come closer to the what the wireframes suggest?

• When making the initial switch from data view mode to SQL mode, a scrollbar that only scrolls ~1px vertically appears for some reason. I recall similar scrollbar issues occurring in fixed viewport apps in the past, so something we're doing here is causing a regression. Would it be possible to investigate and fix?

• Should the Discover tour be updated or otherwise altered when taking the tour in text-based query language mode? Asking because the first step in the tour speaks of using the plus button on the field list to add fields to the table, but I believe that only works for data view mode.

[kertal]

• Should the Discover tour be updated or otherwise altered when taking the tour in text-based query language mode? Asking because the first step in the tour speaks of using the plus button on the field list to add fields to the table, but I believe that only works for data view mode.

@MichaelMarcialis I think it should be hidden for this case

[flash1293]

When making the initial switch from data view mode to SQL mode, a scrollbar that only scrolls ~1px vertically appears for some reason. I recall similar scrollbar issues occurring in fixed viewport apps in the past, so something we're doing here is causing a regression. Would it be possible to investigate and fix?

It's around for a long time, but this hasn't been fixed for Monaco - see #93587

Should the Discover tour be updated or otherwise altered when taking the tour in text-based query language mode? Asking because the first step in the tour speaks of using the plus button on the field list to add fields to the table, but I believe that only works for data view mode.

Agree with @kertal here, let's just hide the tour

[stratoula]

It appears that a custom resizer is being used when the editor is in expanded mode. Would it be possible to switch to using EUI's resizable container components to ensure consistent styling?

I cant use it unfortunately because the code editor lives in the unified search plugin and Discover page is on the discover plugin.

[MichaelMarcialis]

I cant use it unfortunately because the code editor lives in the unified search plugin and Discover page is on the discover plugin.

Forgive the potentially silly question on my part, but could you elaborate on this point? Not sure I'm following on why we can't use the EUI component.

[stratoula]

Of course! If I understand correctly how the EuiResizer works

<EuiResizableContainer direction="vertical">
  {(EuiResizablePanel, EuiResizableButton) => (
    <>
      <EuiResizablePanel initialSize={50} minSize="20%">
        <EuiText>
          <p>{text}</p>
        </EuiText>
      </EuiResizablePanel>

      <EuiResizableButton />

      <EuiResizablePanel initialSize={50} minSize="20%">
        <EuiText>
          <p>{text}</p>
        </EuiText>
      </EuiResizablePanel>
    </>
  )}
</EuiResizableContainer>

you need to wrap the CodeEditor and the rest of the page into the <EuiResizablePanel , right?
These 2 components are not on the same plugin. So I cant think of a way to do it. I could use the EuiResizableButton in order to give the same look and feel but unfortunately it is not exported from EUI.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 0d205c6
• Storybooks Preview

Failed CI Steps

• FTR Configs #22

Test Failures

• [job] [logs] FTR Configs #22 / apis uptime feature controls spaces "after all" hook for "user_1 can't access APIs in space_2"
• [job] [logs] FTR Configs #22 / apis uptime feature controls spaces "before all" hook for "user_1 can access APIs in space_1"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
apm	1302	1303	+1
canvas	1141	1142	+1
cloudSecurityPosture	269	270	+1
controls	172	173	+1
dashboard	416	417	+1
dashboardEnhanced	93	94	+1
data	514	516	+2
dataViewManagement	175	176	+1
dataVisualizer	365	366	+1
discover	590	593	+3
discoverEnhanced	71	72	+1
fleet	747	748	+1
graph	216	217	+1
infra	952	953	+1
inputControlVis	106	107	+1
lens	930	931	+1
lists	303	304	+1
maps	865	866	+1
ml	1624	1625	+1
monitoring	504	505	+1
observability	480	481	+1
osquery	324	325	+1
securitySolution	3154	3155	+1
stackAlerts	137	138	+1
synthetics	886	887	+1
timelines	252	253	+1
transform	319	320	+1
unifiedSearch	196	212	+16
upgradeAssistant	197	198	+1
visTypeTimelion	106	107	+1
visTypeVega	229	230	+1
visualizations	363	364	+1
total			+50

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/es-query	159	168	+9
data	2402	2405	+3
discover	63	65	+2
unifiedSearch	78	82	+4
total			+18

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
cloudSecurityPosture	219.0KB	219.0KB	+61.0B
controls	429.1KB	429.2KB	+61.0B
dashboard	380.2KB	380.7KB	+531.0B
dataViewManagement	143.6KB	143.7KB	+57.0B
dataVisualizer	545.2KB	545.2KB	+57.0B
discover	492.4KB	497.1KB	+4.6KB
discoverEnhanced	42.6KB	43.2KB	+531.0B
infra	1023.2KB	1023.2KB	+61.0B
lens	1.2MB	1.2MB	+531.0B
ml	3.3MB	3.3MB	+61.0B
transform	383.2KB	383.2KB	+61.0B
unifiedSearch	179.0KB	224.1KB	+45.1KB
visTypeTimelion	107.7KB	107.8KB	+61.0B
visTypeVega	1.7MB	1.7MB	+61.0B
visualizations	240.5KB	240.6KB	+85.0B
total			+51.9KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
module count	4967	4969	+2

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
kibana	578	579	+1
unifiedSearch	13	16	+3
total			+4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	421.9KB	422.8KB	+935.0B
discover	28.7KB	28.9KB	+192.0B
maps	80.5KB	80.5KB	+4.0B
navigation	9.6KB	9.6KB	+56.0B
securitySolution	253.4KB	253.4KB	+61.0B
timelines	258.9KB	259.0KB	+61.0B
unifiedSearch	44.0KB	46.9KB	+2.9KB
total			+4.2KB

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
search	13	14	+1

Unknown metric groups

API count

id	before	after	diff
@kbn/es-query	213	222	+9
data	3075	3078	+3
discover	79	81	+2
unifiedSearch	82	96	+14
total			+28

async chunk count

id	before	after	diff
unifiedSearch	11	14	+3

ESLint disabled line counts

id	before	after	diff
@kbn/es-query	8	9	+1
discover	36	37	+1
unifiedSearch	16	19	+3
total			+5

Total ESLint disabled count

id	before	after	diff
@kbn/es-query	9	10	+1
discover	38	39	+1
unifiedSearch	16	19	+3
total			+5

History

• 💔 Build #58843 failed 95da2b3
• 💔 Build #58677 failed 602f4d2
• 💔 Build #58658 failed f6da616
• 💔 Build #58644 failed 39b27ed
• 💔 Build #58597 failed 0558e23
• 💚 Build #58553 succeeded 9893b54

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream