Skip to content

[Search service] Add enhanced ES search strategy#59224

Merged
lukasolson merged 116 commits intoelastic:masterfrom
lukasolson:enhancedEs
Mar 5, 2020
Merged

[Search service] Add enhanced ES search strategy#59224
lukasolson merged 116 commits intoelastic:masterfrom
lukasolson:enhancedEs

Conversation

@lukasolson
Copy link
Contributor

@lukasolson lukasolson commented Mar 3, 2020

Summary

Resolves #54077.

Adds an enhanced ES search strategy which overrides the default ES search strategy provided by the data plugin search service. It adds support for frozen indices (based on the search:includeFrozen UI setting) and rollups (via setting an isRollup parameter on the request).

Checklist

Delete any items that are not applicable to this PR.

For maintainers

lukasolson and others added 30 commits December 9, 2019 11:16
@lukasolson
Add async search strategy
02c57f3
@lukasolson
Add async search
41b7c6a
@lukasolson
Fix async strategy and add tests
cf5ceb9
@lukasolson
Merge branch 'master' into asyncSearch
5ad5b36
@lukasolson
Move types to separate file
2009343
@lukasolson
Revert changes to demo search
ebebdaf
@lukasolson
Merge branch 'master' into asyncSearch
65c39b6
@lukasolson
Update demo search strategy to use async
5dd876f
@lukasolson
Add async es search strategy
d487812
@lukasolson
Return response as rawResponse
7433aa6
@lukasolson
Poll after initial request
93ea93e
@lukasolson
Merge branch 'master' into asyncSearch
8e8eb7b
@lukasolson
Merge branch 'master' into enhancedEsSearch
f15c4af
@lukasolson
Merge branch 'master' into asyncSearch
50b4b69
@lukasolson
Merge branch 'master' into asyncSearch
2bed4a6
@lukasolson
Merge branch 'asyncSearch' into enhancedEsSearch
b6d9c25
@lukasolson
Add cancellation to search strategies
7260761
@lukasolson
Add tests
be300db
@lukasolson
Merge branch 'master' into asyncSearch
fa02de6
@lukasolson
Merge branch 'asyncSearch' into enhancedEsSearch
4e524d1
@lukasolson
Merge branch 'master' into asyncSearch
7ad5755
@lukasolson
Simplify async search strategy
e4eb651
@lukasolson
Merge branch 'asyncSearch' into enhancedEsSearch
b6a7f5e
@lukasolson
Move loadingCount to search strategy
6188c74
@lukasolson
Merge branch 'loadingCount' into asyncSearch
765ec61
@lukasolson
Merge branch 'asyncSearch' into enhancedEsSearch
b35ca0f
@lukasolson
Update abort controller library
f15f138
@lukasolson
Bootstrap
4d36124
@elasticmachine
Merge branch 'master' into abortController
1943855
@elasticmachine
Merge branch 'master' into asyncSearch
c7e328c
@lukasolson lukasolson added Feature:Search Querying infrastructure in Kibana v8.0.0 Team:AppArch release_note:skip Skip the PR/issue when compiling release notes v7.7.0 labels Mar 3, 2020
@lukasolson lukasolson requested a review from lizozom March 3, 2020 21:30
@lukasolson lukasolson requested a review from a team as a code owner March 3, 2020 21:30
@lukasolson lukasolson self-assigned this Mar 3, 2020
@elasticmachine
Copy link
Contributor

elasticmachine commented Mar 3, 2020

Pinging @elastic/kibana-app-arch (Team:AppArch)

@lukasolson lukasolson mentioned this pull request Mar 3, 2020
Merged
5 tasks
lizozom
lizozom approved these changes Mar 5, 2020
View reviewed changes
Copy link
Contributor

@lizozom lizozom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was already able to use this PR, remove the legacy rollup strategy on top of it and test using both regular and rollup index patterns.
LGTM!

src/plugins/data/server/index.ts
export * from './search';
export {
ISearch,
ICancel,
Copy link
Contributor

@lizozom lizozom Mar 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ICancel sounds a bit too generic in the context of the data plugin.
I thing all of these types should have the ISearch prefix

Copy link
Contributor Author

@lukasolson lukasolson Mar 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ We can address this in a follow-up PR

@lizozom lizozom mentioned this pull request Mar 5, 2020
Merged
7 tasks
@kibanamachine
Copy link
Contributor

kibanamachine commented Mar 5, 2020

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@lukasolson lukasolson merged commit c4b385d into elastic:master Mar 5, 2020
jloleysens added a commit to jloleysens/kibana that referenced this pull request Mar 6, 2020
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into ua/fix/reinde…
d73316a 
…x-closed-index

* 'master' of github.com:elastic/kibana: (32 commits)
  [ML] Use Kibana's HttpHandler for HTTP requests (elastic#59320)
  [APM] Create settings page to manage Custom Links (elastic#57788)
  [Upgrade Assistant] Server-side batch reindexing (elastic#58598)
  completes navigation test (elastic#59141)
  [SIEM] Fixes dragging entries to the Timeline while data is loading may trigger a partial page reload (elastic#59476)
  [Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job (elastic#58683)
  [SIEM] [CASES] API with io-ts validation (elastic#59265)
  Use camelCase rather than snakeCase for plugin name (elastic#59461)
  [Maps] top term percentage field property (elastic#59386)
  Add custom action to registry and show actions list in siem (elastic#58395)
  [Search service] Add enhanced ES search strategy (elastic#59224)
  [Logs UI] Speed up stream rendering using memoization (elastic#59163)
  expand max-old-space-size for xpack jest tests (elastic#59455)
  Added possibility to embed connectors create and edit flyouts (elastic#58514)
  Revert "Temporarily disabling PR project mappings (elastic#59485)" (elastic#59491)
  Temporarily disabling PR project mappings (elastic#59485)
  [Endpoint] Fix alert list functional test error (elastic#59357)
  Rename status_page to statusPage (elastic#59186)
  Fix visual baseline job (elastic#59348)
  Extended AlertContextValue with metadata optional property (elastic#59391)
  ...

# Conflicts:
#	x-pack/plugins/upgrade_assistant/common/types.ts
#	x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_actions.ts
#	x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts
#	x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts
#	x-pack/plugins/upgrade_assistant/server/routes/reindex_indices/reindex_indices.test.ts
#	x-pack/plugins/upgrade_assistant/server/routes/reindex_indices/reindex_indices.ts
lukasolson added a commit to lukasolson/kibana that referenced this pull request Mar 6, 2020
@lukasolson @elasticmachine
[Search service] Add enhanced ES search strategy (elastic#59224)
b225e42 
* Add async search strategy

* Add async search

* Fix async strategy and add tests

* Move types to separate file

* Revert changes to demo search

* Update demo search strategy to use async

* Add async es search strategy

* Return response as rawResponse

* Poll after initial request

* Add cancellation to search strategies

* Add tests

* Simplify async search strategy

* Move loadingCount to search strategy

* Update abort controller library

* Bootstrap

* Abort when the request is aborted

* Add utility and update value suggestions route

* Fix bad merge conflict

* Update tests

* Move to data_enhanced plugin

* Remove bad merge

* Revert switching abort controller libraries

* Revert package.json in lib

* Move to previous abort controller

* Add support for frozen indices

* Fix test to use fake timers to run debounced handlers

* Revert changes to example plugin

* Fix loading bar not going away when cancelling

* Call getSearchStrategy instead of passing  directly

* Add async demo search strategy

* Fix error with setting state

* Update how aborting works

* Fix type checks

* Add test for loading count

* Attempt to fix broken example test

* Revert changes to test

* Fix test

* Update name to camelCase

* Fix failing test

* Don't require data_enhanced in example plugin

* Actually send DELETE request

* Use waitForCompletion parameter

* Use default search params

* Add support for rollups

* Only make changes needed for frozen indices/rollups

* Fix tests/types

* Don't include skipped in loaded/total

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@lukasolson lukasolson mentioned this pull request Mar 6, 2020
Merged
@kibanamachine
Copy link
Contributor

kibanamachine commented Mar 6, 2020

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 6, 2020
lukasolson added a commit that referenced this pull request Mar 6, 2020
@lukasolson @elasticmachine
[Search service] Add enhanced ES search strategy (#59224) (#59563)
322a992 
* Add async search strategy

* Add async search

* Fix async strategy and add tests

* Move types to separate file

* Revert changes to demo search

* Update demo search strategy to use async

* Add async es search strategy

* Return response as rawResponse

* Poll after initial request

* Add cancellation to search strategies

* Add tests

* Simplify async search strategy

* Move loadingCount to search strategy

* Update abort controller library

* Bootstrap

* Abort when the request is aborted

* Add utility and update value suggestions route

* Fix bad merge conflict

* Update tests

* Move to data_enhanced plugin

* Remove bad merge

* Revert switching abort controller libraries

* Revert package.json in lib

* Move to previous abort controller

* Add support for frozen indices

* Fix test to use fake timers to run debounced handlers

* Revert changes to example plugin

* Fix loading bar not going away when cancelling

* Call getSearchStrategy instead of passing  directly

* Add async demo search strategy

* Fix error with setting state

* Update how aborting works

* Fix type checks

* Add test for loading count

* Attempt to fix broken example test

* Revert changes to test

* Fix test

* Update name to camelCase

* Fix failing test

* Don't require data_enhanced in example plugin

* Actually send DELETE request

* Use waitForCompletion parameter

* Use default search params

* Add support for rollups

* Only make changes needed for frozen indices/rollups

* Fix tests/types

* Don't include skipped in loaded/total

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@lizozom lizozom lizozom approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@lukasolson lukasolson

Labels

backported Feature:Search Querying infrastructure in Kibana release_note:skip Skip the PR/issue when compiling release notes review v7.7.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Search service] Add support for frozen indices and rollups

4 participants

@lukasolson @elasticmachine @kibanamachine @lizozom

Comments