[Lens] Fix runtime mappings by using new API #83350

wylieconlon · 2020-11-12T21:53:46Z

Instead of removing this code, we can use the new API that was introduced here: elastic/elasticsearch#62906

Another option is to remove the runtime fields entirely, but it appears that we have usage in both Lens and the Security solution.

elasticmachine · 2020-11-12T21:53:48Z

Pinging @elastic/kibana-app (Team:KibanaApp)

flash1293 · 2020-11-16T09:02:55Z

@elasticmachine merge upstream

flash1293 · 2020-11-16T09:04:13Z

@wylieconlon This addition looks suspicious: https://github.com/elastic/kibana/pull/83350/files#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8R3

Do we actually need it?

wylieconlon · 2020-11-16T16:08:19Z

@flash1293 This PR is being tested against a snapshot that isn't available on master, it needs that one-line change to the Jenkinsfile to test the right thing. Will remove before merging.

ghost · 2020-11-16T16:18:07Z

💚 Build Succeeded

pipeline
Commit: d39982e

tylersmalley · 2020-11-17T00:30:24Z

x-pack/test/functional/es_archives/logstash_functional/mappings.json

-          "script" : { "source" : "emit(doc['bytes'].value)" }
+          "type": "long",
+          "script": {
+            "script": {


Should these be double nested?

kibanamachine · 2020-11-17T17:46:36Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: edb00b0
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

X-Pack API Integration Tests (Security Basic).x-pack/test/api_integration/apis/security/index_fields·ts.security (basic license) Index Fields GET /internal/security/fields/{query} should not include runtime fields

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://dryrun

[00:00:00]       │
[00:00:00]         └-: security (basic license)
[00:00:00]           └-> "before all" hook
[00:00:16]           └-: Index Fields
[00:00:16]             └-> "before all" hook
[00:00:16]             └-> "before all" hook
[00:00:16]               │ info [security/flstest/data] Loading "mappings.json"
[00:00:16]               │ info [security/flstest/data] Loading "data.json.gz"
[00:00:16]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1605631580506800623] [flstest] creating index, cause [api], templates [], shards [5]/[1]
[00:00:16]               │ info [security/flstest/data] Created index "flstest"
[00:00:16]               │ debg [security/flstest/data] "flstest" settings {"index":{"number_of_replicas":"1","number_of_shards":"5"}}
[00:00:16]               │ info [security/flstest/data] Indexed 2 docs into "flstest"
[00:00:16]             └-: GET /internal/security/fields/{query}
[00:00:16]               └-> "before all" hook
[00:00:16]               └-> should return a list of available index mapping fields
[00:00:16]                 └-> "before each" hook: global before each
[00:00:16]                 └- ✓ pass  (51ms) "security (basic license) Index Fields GET /internal/security/fields/{query} should return a list of available index mapping fields"
[00:00:16]               └-> should not include runtime fields
[00:00:16]                 └-> "before each" hook: global before each
[00:00:16]                 └- ✖ fail: security (basic license) Index Fields GET /internal/security/fields/{query} should not include runtime fields
[00:00:16]                 │      Error: expected [ '_routing',
[00:00:16]                 │   'customer_name.keyword',
[00:00:16]                 │   '_doc_count',
[00:00:16]                 │   'customer_ssn',
[00:00:16]                 │   '_index',
[00:00:16]                 │   '_feature',
[00:00:16]                 │   'customer_region',
[00:00:16]                 │   '_ignored',
[00:00:16]                 │   '_seq_no',
[00:00:16]                 │   'customer_region.keyword',
[00:00:16]                 │   '_nested_path',
[00:00:16]                 │   '_data_stream_timestamp',
[00:00:16]                 │   '_field_names',
[00:00:16]                 │   '_source',
[00:00:16]                 │   '_id',
[00:00:16]                 │   'customer_name',
[00:00:16]                 │   'customer_ssn.keyword',
[00:00:16]                 │   '_version' ] to contain 'runtime_customer_ssn'
[00:00:16]                 │       at Assertion.assert (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:100:11)
[00:00:16]                 │       at Assertion.contain (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:447:10)
[00:00:16]                 │       at Context.<anonymous> (test/api_integration/apis/security/index_fields.ts:65:63)
[00:00:16]                 │       at processTicksAndRejections (internal/process/task_queues.js:97:5)
[00:00:16]                 │       at Object.apply (/dev/shm/workspace/parallel/3/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:00:16]                 │ 
[00:00:16]                 │

Stack Trace

Error: expected [ '_routing',
  'customer_name.keyword',
  '_doc_count',
  'customer_ssn',
  '_index',
  '_feature',
  'customer_region',
  '_ignored',
  '_seq_no',
  'customer_region.keyword',
  '_nested_path',
  '_data_stream_timestamp',
  '_field_names',
  '_source',
  '_id',
  'customer_name',
  'customer_ssn.keyword',
  '_version' ] to contain 'runtime_customer_ssn'
    at Assertion.assert (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.contain (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:447:10)
    at Context.<anonymous> (test/api_integration/apis/security/index_fields.ts:65:63)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Object.apply (/dev/shm/workspace/parallel/3/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

X-Pack API Integration Tests (Security Basic).x-pack/test/api_integration/apis/security/index_fields·ts.security (basic license) Index Fields GET /internal/security/fields/{query} should not include runtime fields

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: security (basic license)
[00:00:00]           └-> "before all" hook
[00:00:16]           └-: Index Fields
[00:00:16]             └-> "before all" hook
[00:00:16]             └-> "before all" hook
[00:00:16]               │ info [security/flstest/data] Loading "mappings.json"
[00:00:16]               │ info [security/flstest/data] Loading "data.json.gz"
[00:00:16]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xxl-1605631580506800623] [flstest] creating index, cause [api], templates [], shards [5]/[1]
[00:00:16]               │ info [security/flstest/data] Created index "flstest"
[00:00:16]               │ debg [security/flstest/data] "flstest" settings {"index":{"number_of_replicas":"1","number_of_shards":"5"}}
[00:00:16]               │ info [security/flstest/data] Indexed 2 docs into "flstest"
[00:00:16]             └-: GET /internal/security/fields/{query}
[00:00:16]               └-> "before all" hook
[00:00:16]               └-> should return a list of available index mapping fields
[00:00:16]                 └-> "before each" hook: global before each
[00:00:16]                 └- ✓ pass  (48ms) "security (basic license) Index Fields GET /internal/security/fields/{query} should return a list of available index mapping fields"
[00:00:16]               └-> should not include runtime fields
[00:00:16]                 └-> "before each" hook: global before each
[00:00:16]                 └- ✖ fail: security (basic license) Index Fields GET /internal/security/fields/{query} should not include runtime fields
[00:00:16]                 │      Error: expected [ '_routing',
[00:00:16]                 │   'customer_name.keyword',
[00:00:16]                 │   '_doc_count',
[00:00:16]                 │   'customer_ssn',
[00:00:16]                 │   '_index',
[00:00:16]                 │   '_feature',
[00:00:16]                 │   'customer_region',
[00:00:16]                 │   '_ignored',
[00:00:16]                 │   '_seq_no',
[00:00:16]                 │   'customer_region.keyword',
[00:00:16]                 │   '_nested_path',
[00:00:16]                 │   '_data_stream_timestamp',
[00:00:16]                 │   '_field_names',
[00:00:16]                 │   '_source',
[00:00:16]                 │   '_id',
[00:00:16]                 │   'customer_name',
[00:00:16]                 │   'customer_ssn.keyword',
[00:00:16]                 │   '_version' ] to contain 'runtime_customer_ssn'
[00:00:16]                 │       at Assertion.assert (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:100:11)
[00:00:16]                 │       at Assertion.contain (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:447:10)
[00:00:16]                 │       at Context.<anonymous> (test/api_integration/apis/security/index_fields.ts:65:63)
[00:00:16]                 │       at processTicksAndRejections (internal/process/task_queues.js:97:5)
[00:00:16]                 │       at Object.apply (/dev/shm/workspace/parallel/3/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:00:16]                 │ 
[00:00:16]                 │

Stack Trace

Error: expected [ '_routing',
  'customer_name.keyword',
  '_doc_count',
  'customer_ssn',
  '_index',
  '_feature',
  'customer_region',
  '_ignored',
  '_seq_no',
  'customer_region.keyword',
  '_nested_path',
  '_data_stream_timestamp',
  '_field_names',
  '_source',
  '_id',
  'customer_name',
  'customer_ssn.keyword',
  '_version' ] to contain 'runtime_customer_ssn'
    at Assertion.assert (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.contain (/dev/shm/workspace/parallel/3/kibana/packages/kbn-expect/expect.js:447:10)
    at Context.<anonymous> (test/api_integration/apis/security/index_fields.ts:65:63)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Object.apply (/dev/shm/workspace/parallel/3/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

Metrics [docs]

✅ unchanged

History

💔 Build #88113 failed 51c70fb
💔 Build #88041 failed d39982e
💔 Build #87917 failed c8be3e5
💔 Build #87762 failed 803013a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

wylieconlon · 2020-11-17T20:28:16Z

Fixed in #83530

Remove runtime fields from tests

740903d

wylieconlon added Team:Visualizations Team label for Lens, elastic-charts, Graph, legacy editors (TSVB, Visualize, Timelion) t// v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:Lens v7.11.0 labels Nov 12, 2020

Use snapshot to verify test

803013a

Merge branch 'master' into fix-runtime-fields

c8be3e5

wylieconlon requested a review from tylersmalley November 16, 2020 16:06

Remove another bad mapping

d39982e

Use new runtime section in mapping

51c70fb

wylieconlon changed the title ~~[Lens] Remove runtime fields from tests~~ [Lens] Fix runtime mappings by using new API Nov 16, 2020

tylersmalley reviewed Nov 17, 2020

View reviewed changes

Fix mappings

edb00b0

wylieconlon closed this Nov 17, 2020

wylieconlon deleted the fix-runtime-fields branch November 17, 2020 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Lens] Fix runtime mappings by using new API #83350

[Lens] Fix runtime mappings by using new API #83350

Uh oh!

wylieconlon commented Nov 12, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Nov 12, 2020

Uh oh!

flash1293 commented Nov 16, 2020

Uh oh!

flash1293 commented Nov 16, 2020

Uh oh!

wylieconlon commented Nov 16, 2020

Uh oh!

ghost commented Nov 16, 2020

Uh oh!

tylersmalley Nov 17, 2020

Uh oh!

kibanamachine commented Nov 17, 2020 •

edited

Loading

Standard Out

Stack Trace

Standard Out

Stack Trace

Uh oh!

wylieconlon commented Nov 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Lens] Fix runtime mappings by using new API #83350

[Lens] Fix runtime mappings by using new API #83350

Uh oh!

Conversation

wylieconlon commented Nov 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Nov 12, 2020

Uh oh!

flash1293 commented Nov 16, 2020

Uh oh!

flash1293 commented Nov 16, 2020

Uh oh!

wylieconlon commented Nov 16, 2020

Uh oh!

ghost commented Nov 16, 2020

💚 Build Succeeded

Uh oh!

tylersmalley Nov 17, 2020

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Nov 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Metrics [docs]

History

Uh oh!

wylieconlon commented Nov 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wylieconlon commented Nov 12, 2020 •

edited

Loading

kibanamachine commented Nov 17, 2020 •

edited

Loading