Skip to content

[7.x] [Snapshot and Restore] Server side snapshots pagination (#110266)#115434

Merged
kibanamachine merged 5 commits intoelastic:7.xfrom
kibanamachine:backport/7.x/pr-110266
Oct 19, 2021
Merged

[7.x] [Snapshot and Restore] Server side snapshots pagination (#110266)#115434
kibanamachine merged 5 commits intoelastic:7.xfrom
kibanamachine:backport/7.x/pr-110266

Conversation

@kibanamachine
Copy link
Contributor

@kibanamachine kibanamachine commented Oct 18, 2021

Backports the following commits to 7.x:

@yuliacech @kibanamachine
@cjcenizal @jrodewig
[Snapshot and Restore] Server side snapshots pagination (elastic#110266)
50b611f 
* [Snapshot % Restore] Added server side pagination and sorting to get snapshots route, refactored snapshots table to use EuiBasicTable with controlled pagination instead of EuiInMemoryTable

* [Snapshot & Restore] Added server side sorting by shards and failed shards counts

* [Snapshot & Restore] Fixed i18n errors

* [Snapshot & Restore] Added server side sorting by repository

* [Snapshot & Restore] Implemented server side search request for snapshot, repository and policy name

* [Snapshot & Restore] Fixed eslint errors

* [Snapshot & Restore] Removed uncommented code

* [Snapshot & Restore] Fixed pagination/search bug

* [Snapshot & Restore] Fixed pagination/search bug

* [Snapshot & Restore] Fixed text truncate bug

* [Snapshot & Restore] Fixed non existent repository search error

* Update x-pack/plugins/snapshot_restore/public/application/sections/home/snapshot_list/components/snapshot_search_bar.tsx

Co-authored-by: CJ Cenizal <cj@cenizal.com>

* Update x-pack/plugins/snapshot_restore/public/application/sections/home/snapshot_list/components/snapshot_empty_prompt.tsx

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

* [Snapshot & Restore] Fixed missing i18n and no snapshots callout

* [Snapshot & Restore] Moved "getSnapshotSearchWildcard" to a separate file and added unit tests

* [Snapshot & Restore] Added api integration tests for "get snapshots" endpoint (pagination, sorting, search)

* [Snapshot & Restore] Renamed SnapshotSearchOptions/SnapshotTableOptions to -Params and added the link to the specs issue

* [Snapshot & Restore] Fixed search wildcard to also match string in the middle of the value, not only starting with the string. Also updated the tests following the code review suggestions to make them more readable.

* [Snapshot & Restore] Added incremental search back to snapshots list and a debounce of 500ms

* [Snapshot & Restore] Updated snapshot search debounce value and extracted it into a constant

* [Snapshot & Restore] Renamed debounceValue to cachedListParams and added a comment why debounce is used

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
@kibanamachine kibanamachine added the backport This PR is a backport of another PR label Oct 18, 2021
@kibanamachine kibanamachine enabled auto-merge (squash) October 18, 2021 18:59
@kibanamachine kibanamachine mentioned this pull request Oct 18, 2021
Merged
4 tasks
@yuliacech
Copy link
Contributor

yuliacech commented Oct 18, 2021

@elasticmachine merge upstream

@yuliacech
Copy link
Contributor

yuliacech commented Oct 19, 2021

@elasticmachine merge upstream

@yuliacech
Copy link
Contributor

yuliacech commented Oct 19, 2021

@elasticmachine merge upstream

kibanamachine and others added 2 commits October 19, 2021 07:59
@kibanamachine
Merge branch '7.x' into backport/7.x/pr-110266
0673cfd
@yuliacech
Updated config.ts to fix UA test
ea1370a 
UA functional API integration test to check cloud backup status creates a snapshot repo, which fails to be created with my changes to config.ts `'path.repo=/tmp/repo,/tmp/repo_1,/tmp/repo_2,'`. Adding `/tmp/cloud-snapshots/'` to the config fixes the test.
@kibanamachine
Copy link
Contributor Author

kibanamachine commented Oct 19, 2021

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/ml/jobs/categorization_field_examples·ts.apis Machine Learning jobs Categorization example endpoint - partially valid, more than 75% are null

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]     │
[00:00:00]       └-: apis
[00:00:00]         └-> "before all" hook in "apis"
[00:09:34]         └-: Machine Learning
[00:09:34]           └-> "before all" hook in "Machine Learning"
[00:09:34]           └-> "before all" hook in "Machine Learning"
[00:09:34]             │ debg creating role ft_ml_source
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_source]
[00:09:34]             │ debg creating role ft_ml_source_readonly
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_source_readonly]
[00:09:34]             │ debg creating role ft_ml_dest
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_dest]
[00:09:34]             │ debg creating role ft_ml_dest_readonly
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_dest_readonly]
[00:09:34]             │ debg creating role ft_ml_ui_extras
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_ui_extras]
[00:09:34]             │ debg creating role ft_default_space_ml_all
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space_ml_all]
[00:09:34]             │ debg creating role ft_default_space1_ml_all
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space1_ml_all]
[00:09:34]             │ debg creating role ft_all_spaces_ml_all
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_all_spaces_ml_all]
[00:09:34]             │ debg creating role ft_default_space_ml_read
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space_ml_read]
[00:09:34]             │ debg creating role ft_default_space1_ml_read
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space1_ml_read]
[00:09:34]             │ debg creating role ft_all_spaces_ml_read
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_all_spaces_ml_read]
[00:09:34]             │ debg creating role ft_default_space_ml_none
[00:09:34]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space_ml_none]
[00:09:34]             │ debg creating user ft_ml_poweruser
[00:09:34]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser]
[00:09:34]             │ debg created user ft_ml_poweruser
[00:09:34]             │ debg creating user ft_ml_poweruser_spaces
[00:09:34]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser_spaces]
[00:09:34]             │ debg created user ft_ml_poweruser_spaces
[00:09:34]             │ debg creating user ft_ml_poweruser_space1
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser_space1]
[00:09:35]             │ debg created user ft_ml_poweruser_space1
[00:09:35]             │ debg creating user ft_ml_poweruser_all_spaces
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser_all_spaces]
[00:09:35]             │ debg created user ft_ml_poweruser_all_spaces
[00:09:35]             │ debg creating user ft_ml_viewer
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer]
[00:09:35]             │ debg created user ft_ml_viewer
[00:09:35]             │ debg creating user ft_ml_viewer_spaces
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer_spaces]
[00:09:35]             │ debg created user ft_ml_viewer_spaces
[00:09:35]             │ debg creating user ft_ml_viewer_space1
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer_space1]
[00:09:35]             │ debg created user ft_ml_viewer_space1
[00:09:35]             │ debg creating user ft_ml_viewer_all_spaces
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer_all_spaces]
[00:09:35]             │ debg created user ft_ml_viewer_all_spaces
[00:09:35]             │ debg creating user ft_ml_unauthorized
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_unauthorized]
[00:09:35]             │ debg created user ft_ml_unauthorized
[00:09:35]             │ debg creating user ft_ml_unauthorized_spaces
[00:09:35]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_unauthorized_spaces]
[00:09:35]             │ debg created user ft_ml_unauthorized_spaces
[00:13:51]           └-: jobs
[00:13:51]             └-> "before all" hook in "jobs"
[00:13:51]             └-: Categorization example endpoint - 
[00:13:51]               └-> "before all" hook for "valid with good number of tokens"
[00:13:51]               └-> "before all" hook for "valid with good number of tokens"
[00:13:51]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Loading "mappings.json"
[00:13:51]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Loading "data.json.gz"
[00:13:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [ft_categorization] creating index, cause [api], templates [], shards [1]/[0]
[00:13:51]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Created index "ft_categorization"
[00:13:51]                 │ debg [x-pack/test/functional/es_archives/ml/categorization] "ft_categorization" settings {"index":{"number_of_replicas":"0","number_of_shards":"1"}}
[00:13:52]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Indexed 1501 docs into "ft_categorization"
[00:13:52]                 │ debg applying update to kibana config: {"dateFormat:tz":"UTC"}
[00:13:52]               └-> valid with good number of tokens
[00:13:52]                 └-> "before each" hook: global before each for "valid with good number of tokens"
[00:13:53]                 └- ✓ pass  (134ms)
[00:13:53]               └-> invalid, too many tokens.
[00:13:53]                 └-> "before each" hook: global before each for "invalid, too many tokens."
[00:13:53]                 │ info [r.suppressed] [node-01] path: /_analyze, params: {}
[00:13:53]                 │      org.elasticsearch.transport.RemoteTransportException: [node-01][127.0.0.1:6341][indices:admin/analyze[s]]
[00:13:53]                 │      Caused by: java.lang.IllegalStateException: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting.
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.increment(TransportAnalyzeAction.java:397) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.access$100(TransportAnalyzeAction.java:387) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.simpleAnalyze(TransportAnalyzeAction.java:229) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:204) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:122) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:110) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:62) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:99) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[00:13:53]                 │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[00:13:53]                 │      	at java.lang.Thread.run(Thread.java:833) [?:?]
[00:13:53]                 │ info [r.suppressed] [node-01] path: /_analyze, params: {}
[00:13:53]                 │      org.elasticsearch.transport.RemoteTransportException: [node-01][127.0.0.1:6341][indices:admin/analyze[s]]
[00:13:53]                 │      Caused by: java.lang.IllegalStateException: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting.
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.increment(TransportAnalyzeAction.java:397) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.access$100(TransportAnalyzeAction.java:387) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.simpleAnalyze(TransportAnalyzeAction.java:229) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:204) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:122) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:110) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:62) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:99) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:53]                 │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[00:13:53]                 │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[00:13:53]                 │      	at java.lang.Thread.run(Thread.java:833) [?:?]
[00:13:53]                 └- ✓ pass  (249ms)
[00:13:53]               └-> partially valid, more than 75% are null
[00:13:53]                 └-> "before each" hook: global before each for "partially valid, more than 75% are null"
[00:13:53]                 └- ✖ fail: apis Machine Learning jobs Categorization example endpoint -  partially valid, more than 75% are null
[00:13:53]                 │       Error: expected 249 to sort of equal 250
[00:13:53]                 │       + expected - actual
[00:13:53]                 │ 
[00:13:53]                 │       -249
[00:13:53]                 │       +250
[00:13:53]                 │       
[00:13:53]                 │       at Assertion.assert (/dev/shm/workspace/parallel/4/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:13:53]                 │       at Assertion.eql (/dev/shm/workspace/parallel/4/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:13:53]                 │       at Context.<anonymous> (test/api_integration/apis/ml/jobs/categorization_field_examples.ts:303:36)
[00:13:53]                 │       at runMicrotasks (<anonymous>)
[00:13:53]                 │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
[00:13:53]                 │       at Object.apply (/dev/shm/workspace/parallel/4/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:13:53]                 │ 
[00:13:53]                 │

Stack Trace

Error: expected 249 to sort of equal 250
    at Assertion.assert (/dev/shm/workspace/parallel/4/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/4/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/ml/jobs/categorization_field_examples.ts:303:36)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.apply (/dev/shm/workspace/parallel/4/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16) {
  actual: '249',
  expected: '250',
  showDiff: true
}

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
snapshotRestore 182 189 +7

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
snapshotRestore 256.9KB 259.0KB +2.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
snapshotRestore 28.7KB 29.0KB +351.0B
Unknown metric groups

API count

id before after diff
snapshotRestore 23 22 -1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @yuliacech

@kibanamachine kibanamachine merged commit c38e3cc into elastic:7.x Oct 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@yuliacech yuliacech

Labels

backport This PR is a backport of another PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kibanamachine @yuliacech