Skip to content

[7.x] [bfetch] compress stream chunks (#97994)#101027

Merged
lizozom merged 1 commit intoelastic:7.xfrom
lizozom:backport/7.x/pr-97994
Jun 1, 2021
Merged

[7.x] [bfetch] compress stream chunks (#97994)#101027
lizozom merged 1 commit intoelastic:7.xfrom
lizozom:backport/7.x/pr-97994

Conversation

@lizozom
Copy link
Copy Markdown
Contributor

@lizozom lizozom commented Jun 1, 2021

Backports the following commits to 7.x:

* Move inspector adapter integration into search source

* docs and ts

* Move other bucket to search source

* test ts + delete unused tabilfy function

* hierarchical param in aggconfig.
ts improvements
more inspector tests

* fix jest

* separate inspect
more tests

* jest

* inspector

* Error handling and more tests

* put the fun in functional tests

* delete client side legacy msearch code

* ts

* override to sync search in search source

* delete more legacy code

* ts

* delete moarrrr

* deflate bfetch chunks

* update tests
use only zlib

* ts

* extract getInflatedResponse

* tests

* Use fflate in attempt to reduce package size

* use node streams, fflate and hex encoding.

* DISABLE_SEARCH_COMPRESSION UI Settings
Use base64 and async compression

* i18n

* Code review
Use custom header for compression
Promisify once

* use custom headers

* Update jest

* fix tests

* code review, baby!

* integration

* tests

* limit

* limit

* limit

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
#	packages/kbn-optimizer/limits.yml
@lizozom lizozom requested review from a team as code owners June 1, 2021 07:10
@lizozom lizozom added the backport This PR is a backport of another PR label Jun 1, 2021
@lizozom lizozom enabled auto-merge (squash) June 1, 2021 07:10
@kibanamachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/management/index_management/indices·js.apis management index management indices reload (not on Cloud) should list all the indices with the expected properties and data enrichers

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 5 times on tracked branches: https://github.com/elastic/kibana/issues/90565

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:05:22]           └-: management
[00:05:22]             └-> "before all" hook in "management"
[00:05:41]             └-: index management
[00:05:41]               └-> "before all" hook in "index management"
[00:05:41]               └-: indices
[00:05:41]                 └-> "before all" hook in "indices"
[00:05:42]                 └-: reload
[00:05:42]                   └-> "before all" hook for "should allow reloading only certain indices"
[00:05:42]                   └-> should allow reloading only certain indices
[00:05:42]                     └-> "before each" hook: global before each for "should allow reloading only certain indices"
[00:05:42]                     │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1622531490023449040] [ximsqwh-1622534601250] creating index, cause [api], templates [], shards [1]/[1]
[00:05:43]                     └- ✓ pass  (67ms) "apis management index management indices reload should allow reloading only certain indices"
[00:05:43]                   └-: (not on Cloud)
[00:05:43]                     └-> "before all" hook for "should list all the indices with the expected properties and data enrichers"
[00:05:43]                     └-> should list all the indices with the expected properties and data enrichers
[00:05:43]                       └-> "before each" hook: global before each for "should list all the indices with the expected properties and data enrichers"
[00:05:43]                       └- ✖ fail: apis management index management indices reload (not on Cloud) should list all the indices with the expected properties and data enrichers
[00:05:43]                       │       Error: expected [ 'aliases',
[00:05:43]                       │   'data_stream',
[00:05:43]                       │   'documents',
[00:05:43]                       │   'health',
[00:05:43]                       │   'hidden',
[00:05:43]                       │   'ilm',
[00:05:43]                       │   'isFollowerIndex',
[00:05:43]                       │   'isFrozen',
[00:05:43]                       │   'isRollupIndex',
[00:05:43]                       │   'name',
[00:05:43]                       │   'primary',
[00:05:43]                       │   'replica',
[00:05:43]                       │   'size',
[00:05:43]                       │   'status',
[00:05:43]                       │   'uuid' ] to sort of equal [ 'aliases',
[00:05:43]                       │   'documents',
[00:05:43]                       │   'health',
[00:05:43]                       │   'hidden',
[00:05:43]                       │   'ilm',
[00:05:43]                       │   'isFollowerIndex',
[00:05:43]                       │   'isFrozen',
[00:05:43]                       │   'isRollupIndex',
[00:05:43]                       │   'name',
[00:05:43]                       │   'primary',
[00:05:43]                       │   'replica',
[00:05:43]                       │   'size',
[00:05:43]                       │   'status',
[00:05:43]                       │   'uuid' ]
[00:05:43]                       │       + expected - actual
[00:05:43]                       │ 
[00:05:43]                       │        [
[00:05:43]                       │          "aliases"
[00:05:43]                       │       -  "data_stream"
[00:05:43]                       │          "documents"
[00:05:43]                       │          "health"
[00:05:43]                       │          "hidden"
[00:05:43]                       │          "ilm"
[00:05:43]                       │       
[00:05:43]                       │       at Assertion.assert (/dev/shm/workspace/parallel/2/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:05:43]                       │       at Assertion.eql (/dev/shm/workspace/parallel/2/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:05:43]                       │       at Context.<anonymous> (test/api_integration/apis/management/index_management/indices.js:237:41)
[00:05:43]                       │       at Object.apply (/dev/shm/workspace/parallel/2/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:05:43]                       │ 
[00:05:43]                       │ 

Stack Trace

Error: expected [ 'aliases',
  'data_stream',
  'documents',
  'health',
  'hidden',
  'ilm',
  'isFollowerIndex',
  'isFrozen',
  'isRollupIndex',
  'name',
  'primary',
  'replica',
  'size',
  'status',
  'uuid' ] to sort of equal [ 'aliases',
  'documents',
  'health',
  'hidden',
  'ilm',
  'isFollowerIndex',
  'isFrozen',
  'isRollupIndex',
  'name',
  'primary',
  'replica',
  'size',
  'status',
  'uuid' ]
    at Assertion.assert (/dev/shm/workspace/parallel/2/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/2/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/management/index_management/indices.js:237:41)
    at Object.apply (/dev/shm/workspace/parallel/2/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16) {
  actual: '[\n' +
    '  "aliases"\n' +
    '  "data_stream"\n' +
    '  "documents"\n' +
    '  "health"\n' +
    '  "hidden"\n' +
    '  "ilm"\n' +
    '  "isFollowerIndex"\n' +
    '  "isFrozen"\n' +
    '  "isRollupIndex"\n' +
    '  "name"\n' +
    '  "primary"\n' +
    '  "replica"\n' +
    '  "size"\n' +
    '  "status"\n' +
    '  "uuid"\n' +
    ']',
  expected: '[\n' +
    '  "aliases"\n' +
    '  "documents"\n' +
    '  "health"\n' +
    '  "hidden"\n' +
    '  "ilm"\n' +
    '  "isFollowerIndex"\n' +
    '  "isFrozen"\n' +
    '  "isRollupIndex"\n' +
    '  "name"\n' +
    '  "primary"\n' +
    '  "replica"\n' +
    '  "size"\n' +
    '  "status"\n' +
    '  "uuid"\n' +
    ']',
  showDiff: true
}

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
bfetch 21 29 +8

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
bfetch 58 62 +4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
bfetch 11.1KB 46.9KB +35.8KB
kbnUiSharedDeps-elastic 3.3MB 3.3MB +1.0B
kbnUiSharedDeps-js 6.8MB 6.8MB +29.5KB
total +65.2KB
Unknown metric groups

API count

id before after diff
bfetch 69 73 +4

References to deprecated APIs

id before after diff
crossClusterReplication 8 6 -2
globalSearch 4 2 -2
indexManagement 12 7 -5
lens 67 45 -22
securitySolution 390 346 -44
stackAlerts 101 95 -6
total -81

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@lizozom lizozom merged commit 157e2cf into elastic:7.x Jun 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants