[Maps] Use default format when proxying EMS-files #79760

thomasneirynck · 2020-10-06T20:35:58Z

NOTE: I added the api-test, just curious to see what it would do in CI. Need to discuss.

Summary

Closes #71975

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-10-06T20:36:24Z

Pinging @elastic/kibana-gis (Team:Geo)

thomasneirynck · 2020-10-06T23:30:38Z

@elasticmachine merge upstream

…a into maps/proxied_format

jsanz · 2020-10-08T11:32:12Z

@thomasneirynck I've checked this branch but I can't start Kibana 😞

I've tested against master and 7.10 and both branches are working on my environment. I also tried to merge this branch against master and then even it run, I couldn't get into the login screen.

These are the errors I got on the console with a clean checkout of this PR branch:

Kibana log

...
  log   [13:24:36.335] [info][server] basepath proxy server running at http://0.0.0.0:5601/noh
server    log   [13:24:49.027] [info][plugins-service] Plugin "visTypeXy" is disabled.
server    log   [13:24:49.038] [info][plugins-service] Plugin "auditTrail" is disabled.
server    log   [13:24:49.579] [info][plugins-system] Setting up [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,esUiShared,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeTable,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeVega,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,bfetch,canvas,charts,visTypeMetric,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]
server    log   [13:24:50.147] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
server    log   [13:24:50.492] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
server    log   [13:24:50.803] [info][monitoring][monitoring][plugins] config sourced from: production cluster
server    log   [13:24:51.287] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [13:24:51.434] [info][savedobjects-service] Starting saved objects migrations
server    log   [13:24:51.559] [info][savedobjects-service] Creating index .kibana_1.
server    log   [13:24:51.605] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server    log   [13:24:51.609] [error][data][elasticsearch] [mapper_parsing_exception]: Failed to parse mapping: unknown parameter [doc_values] on mapper [uiStateJSON] of type [text]
server    log   [13:24:51.611] [warning][savedobjects-service] Unable to connect to Elasticsearch. Error: mapper_parsing_exception
server    log   [13:24:51.612] [fatal][root] { ResponseError: mapper_parsing_exception
    at IncomingMessage.response.on (/home/j/src/elastic/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:257:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 400 },
     statusCode: 400,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '425' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
server    log   [13:24:51.624] [info][plugins-system] Stopping all plugins.
server    log   [13:24:51.779] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server    log   [13:24:51.873] [info][savedobjects-service] Finished in 268ms.

 FATAL  ResponseError: mapper_parsing_exception

 server crashed  with status code 1

Elasticsearch logs

...
   │ info [o.e.c.m.MetadataIndexTemplateService] [a] adding index template [synthetics] for index patterns [synthetics-*-*]
   │ info [o.e.c.r.a.AllocationService] [a] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-7][0]]])." previous.health="YELLOW" reason="shards started [[.security-7][0]]"
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [ml-size-based-ilm-policy]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [metrics]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [logs]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [synthetics]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [watch-history-ilm-policy]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [ilm-history-ilm-policy]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [slm-history-ilm-policy]
   │ info [o.e.l.LicenseService] [a] license [3c7f1e2f-274c-40ab-81a6-383b3caa5fc8] mode [basic] - valid
   │ info [o.e.x.s.s.SecurityStatusChangeListener] [a] Active license is now [BASIC]; Security is enabled
   │ info [o.e.c.m.MetadataCreateIndexService] [a] failed on parsing mappings on index creation [.kibana_1]
   │      org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: unknown parameter [doc_values] on mapper [uiStateJSON] of type [text]
   │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:289) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:262) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.updateIndexMappingsAndBuildSortOrder(MetadataCreateIndexService.java:918) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.lambda$applyCreateIndexWithTemporaryService$2(MetadataCreateIndexService.java:412) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:621) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:410) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:488) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:373) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:380) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:303) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:674) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
   │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
   │      	at java.lang.Thread.run(Thread.java:832) [?:?]
   │      Caused by: org.elasticsearch.index.mapper.MapperParsingException: unknown parameter [doc_values] on mapper [uiStateJSON] of type [text]
   │      	at org.elasticsearch.index.mapper.ParametrizedFieldMapper$Builder.parse(ParametrizedFieldMapper.java:621) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:666) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:651) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:320) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:237) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:209) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:320) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:237) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:111) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:287) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	... 22 more
   │ info [o.e.c.m.MetadataCreateIndexService] [a] [.kibana_task_manager_1] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [a] updating number_of_replicas to [0] for indices [.kibana_task_manager_1]
   │ info [o.e.c.r.a.AllocationService] [a] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_task_manager_1][0]]])." previous.health="YELLOW" reason="shards started [[.kibana_task_manager_1][0]]"

thomasneirynck · 2020-10-08T12:54:51Z

@jsanz I'm seeing this on other PRs as well (e.g. #79934) tbd..

thomasneirynck · 2020-10-08T13:03:39Z

@elasticmachine merge upstream

jsanz

I could test this locally with:

Serve locally a "broken" manifest using the World Countries new data from elastic/ems-file-service#192 and putting the GeoJSON format before the default TopoJSON
Tested that this set up is indeed broken in branch 7.10, where the GeoJSON is pulled but expecting a TopoJSON.
Tested that in this branch the correct dataset is pulled and correctly processed and presented

thomasneirynck · 2020-10-12T15:27:36Z

@elasticmachine merge upstream

nickpeihl

just a couple of nits in tests. but otherwise lgtm. tested proxy with multiple formats and confirmed it works.

wrt testing. rather than use the cloud ems service, I wonder if we shouldn't create a local EMS mock plugin similar to how the newsfeed is mocked for tests? maybe out of scope for this PR though.

x-pack/test/api_integration/apis/maps/proxy_api.js

…format

kibanamachine · 2020-10-26T23:07:30Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d0740e2

Metrics [docs]

✅ unchanged

History

💚 Build #81111 succeeded 5cd0147
💚 Build #80730 succeeded 5d68ad0
💚 Build #80467 succeeded 5664f3c
💚 Build #80348 succeeded 9a47be6
💛 Build #80298 was flaky 6e7a9ad

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: [Security Solution][Endpoint][Admin] Malware Protections Notify User Version (elastic#81415) Revert "[Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81390)" [Maps] Use default format when proxying EMS-files (elastic#79760) [Discover] Deangularize context.html (elastic#81442) Use the displayName property in default editor (elastic#73311) adds bug label to Bug report for Security Solution template (elastic#81643) [ML] Transforms: Remove index field limitation for custom query. (elastic#81467) [Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81390) [Task Manager] Mark task as failed if maxAttempts has been met. (elastic#80681) [Lens] Fix URL query loss on redirect (elastic#81475) Log reason for 404 in field existence check (elastic#81315)

* master: (87 commits) [Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81778) [i18n] add get_kibana_translation_paths tests (elastic#81624) [UX] Fix search term reset from url (elastic#81654) [Lens] Improved range formatter (elastic#80132) [Resolver] `SideEffectContext` changes, remove `@testing-library` uses (elastic#81077) [Time to Visualize] Update Library Text with Call to Action (elastic#81667) [docs] Resolving failed Kibana upgrade migrations (elastic#80999) [ftr/menuToggle] provide helper for enhanced menu toggle handling (elastic#81709) [Task Manager] adds basic observability into Task Manager's runtime operations (elastic#77868) Doc changes for stack management and grouped feature privileges (elastic#80486) Added functional test for alerts list filters by status, alert type and action type. Did a code refactoring and splitting for alerts tests. (elastic#81422) [Security Solution][Endpoint][Admin] Malware Protections Notify User Version (elastic#81415) Revert "[Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81390)" [Maps] Use default format when proxying EMS-files (elastic#79760) [Discover] Deangularize context.html (elastic#81442) Use the displayName property in default editor (elastic#73311) adds bug label to Bug report for Security Solution template (elastic#81643) [ML] Transforms: Remove index field limitation for custom query. (elastic#81467) [Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81390) [Task Manager] Mark task as failed if maxAttempts has been met. (elastic#80681) ...

use default format

d5fc9ab

thomasneirynck requested a review from a team as a code owner October 6, 2020 20:35

thomasneirynck added release_note:fix Team:Geo Former Team Label for Geo Team. Now use Team:Presentation v7.10.0 v7.11.0 v8.0.0 labels Oct 6, 2020

add functional test (tbd)

4bd8d11

thomasneirynck requested review from jsanz and nickpeihl October 6, 2020 20:45

add server flag

6e7a9ad

kibanamachine and others added 3 commits October 6, 2020 19:30

Merge branch 'master' into maps/proxied_format

9a47be6

add api test file

b2d76a3

Merge branch 'maps/proxied_format' of github.com:thomasneirynck/kiban…

5664f3c

…a into maps/proxied_format

Merge branch 'master' into maps/proxied_format

5d68ad0

jsanz approved these changes Oct 8, 2020

View reviewed changes

Merge branch 'master' into maps/proxied_format

5cd0147

nickpeihl approved these changes Oct 12, 2020

View reviewed changes

x-pack/test/api_integration/apis/maps/proxy_api.js Outdated Show resolved Hide resolved

x-pack/test/api_integration/apis/maps/proxy_api.js Outdated Show resolved Hide resolved

thomasneirynck added 2 commits October 26, 2020 17:26

Merge branch 'master' of github.com:elastic/kibana into maps/proxied_…

1cc973a

…format

make tests more robust

d0740e2

thomasneirynck merged commit 2d0106d into elastic:master Oct 27, 2020

thomasneirynck mentioned this pull request Oct 27, 2020

[7.x] [Maps] Use default format when proxying EMS-files (#79760) #81767

Merged

thomasneirynck added a commit to thomasneirynck/kibana that referenced this pull request Oct 27, 2020

[Maps] Use default format when proxying EMS-files (elastic#79760)

041322e

thomasneirynck mentioned this pull request Oct 27, 2020

[7.10] [Maps] Use default format when proxying EMS-files (#79760) #81768

Merged

thomasneirynck added a commit to thomasneirynck/kibana that referenced this pull request Oct 27, 2020

[Maps] Use default format when proxying EMS-files (elastic#79760)

f7d39bc

thomasneirynck added a commit that referenced this pull request Oct 27, 2020

[Maps] Use default format when proxying EMS-files (#79760) (#81768)

0fdcda1

thomasneirynck added a commit that referenced this pull request Oct 27, 2020

[Maps] Use default format when proxying EMS-files (#79760) (#81767)

aee72ff

nreese mentioned this pull request Oct 27, 2020

[Maps] Fix EMS test #81856

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Maps] Use default format when proxying EMS-files #79760

[Maps] Use default format when proxying EMS-files #79760

Uh oh!

thomasneirynck commented Oct 6, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Oct 6, 2020

Uh oh!

thomasneirynck commented Oct 6, 2020

Uh oh!

jsanz commented Oct 8, 2020

Uh oh!

thomasneirynck commented Oct 8, 2020

Uh oh!

thomasneirynck commented Oct 8, 2020

Uh oh!

jsanz left a comment

Uh oh!

thomasneirynck commented Oct 12, 2020

Uh oh!

nickpeihl left a comment

Uh oh!

Uh oh!

Uh oh!

kibanamachine commented Oct 26, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Maps] Use default format when proxying EMS-files #79760

[Maps] Use default format when proxying EMS-files #79760

Uh oh!

Conversation

thomasneirynck commented Oct 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

For maintainers

Uh oh!

elasticmachine commented Oct 6, 2020

Uh oh!

thomasneirynck commented Oct 6, 2020

Uh oh!

jsanz commented Oct 8, 2020

Uh oh!

thomasneirynck commented Oct 8, 2020

Uh oh!

thomasneirynck commented Oct 8, 2020

Uh oh!

jsanz left a comment

Choose a reason for hiding this comment

Uh oh!

thomasneirynck commented Oct 12, 2020

Uh oh!

nickpeihl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kibanamachine commented Oct 26, 2020

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thomasneirynck commented Oct 6, 2020 •

edited

Loading