Elasticsearch 7 Support #831

orangejulius · 2019-10-28T16:30:24Z

This issue will track support for Elasticsearch 7 in Pelias.

Most Elasticsearch upgrades require two sets of changes:

Base compatibility changes, often dropping use of functionality no longer supported by Elasticsearch. These changes are generally required before the new version of Elasticsearch works at all
Tweaks and changes to ensure that queries return the correct results and with adequate performance. These are usually a bit more subjective and can come after initial support has been completed.

Pelias Tasks

Here's the list of breaking changes we'll need to adapt to (this list will be updated over time):

Support running on Elasticsearch 6 as a base step
[illegal_argument_exception] Token filter [word_delimiter] cannot be used to parse synonyms when using pelias/schema (possibly related to Correct our use of synonyms for ES6 schema#381). solved in backport some features from the ES7 branch which may be ok to use now schema#403, another solution proposed in feat(peliasAdmin): Remove word delimiter filter schema#392
Remove use of the doc mapping type in pelias-model and pelias-schema
Remove schema configuration to disable the _all field. The _all field is always disabled in ES6+, so this code isn't needed once we are on ES6. It is incompatible with ES7. (done in feat(es7): remove _all mapping schema#424)

Reference links

The text was updated successfully, but these errors were encountered:

This is the first step in supporting Elasticsearch 7. At this time, Pelias does not work out of the box on ES7, but with a Docker image ready to go, we can begin testing changes for compatibility. This Dockerfile and config is identical to the ES6 Docker image, except for changing the version, and making one update to the `elasticsearch.yml`: In ES7, the bulk thread pool is removed, and both bulk and non-bulk operations go through a single [write](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool) thread pool. For Pelias we have found increasing the queue size of this thread pool is useful to ensure imports can succeed without errors, so the configuration file has been updated accordingly. Connects pelias/pelias#831

This is necessary for Elasticsearch 7 Connects pelias/pelias#831

orangejulius · 2019-11-07T16:31:42Z

With the list of changes above as of this writing, an ES7 build and an import of a few million records for the Portland Metro area works well, and querying with the latest API causes no errors.

I'm sure there's more work to do, in particular I think at least one geo query related change will be required, but it looks like the core part of the ES7 upgrade is now fairly well understood! 🎉

The first error seen when trying to use our current schema with Elasticsearch 7 is: ``` [illegal_argument_exception] Token filter [word_delimiter] cannot be used to parse synonyms ``` The [word delimiter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html) token filter is only used in one place: the `peliasAdmin` analyzer. Looking at the documentation for `word_delimiter`, it does _a lot_: splitting words, handling punctuation, and even some basic stemming. It really feels like an extremely broad tool and at this point feels like something that Elasticsearch would deprecate in the future. Furthermore, looking at our integration tests, it seems one of the key reasons we used it was to tokenize on hyphens, which we have done using the `peliasNameTokenizer` since #375. Considering how complicated this token filter is, and how it's now being used with relatively little effect, it seems like something we can remove. Connects pelias/pelias#831

missinglink · 2019-11-26T13:06:08Z

after merging pelias/schema#403 its now possible to create indices on ES 6.8.5 which will be compatible with 7.4.2

missinglink · 2019-11-27T14:13:06Z

For the adventurous among you, we have a prelease pelias/schema branch here.
You'll find the corresponding docker images here.

At a minimum you should ensure that you've made the following configuration changes for ES7:

Update pelias.json to set the correct esclient.apiVersion (7.4 at time of writing)
Set the schema.typeName property to _doc in pelias.json (note the underscore!)
Update docker-compose.yml to set the correct services.elasticsearch.image (pelias/elasticsearch:7.4.2 at time of writing)

Elasticsearch 6 is now supported, we are working on ES7! Connects pelias/pelias#719 Connects pelias/pelias#831

In Elasticsearch 7+, the [hits count is now an object](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#hits-total-now-object-search-response). This was needed because Elasticsearch now includes a performance improvement that allows non-exact hit counts to be used when the exact count isn't needed. This adds a helper to wrap around the breaking change and support either the old or new format. Extracted from #394 Connects pelias/pelias#831

BREAKING CHANGE This drops Elasticsearch 5 from the test matrix for this repo. While it makes no other changes, it's a breaking change as it marks the point where we stop supporting ES5. This allows us to make changes that support only ES6/7 as we add support for Elasticsearch 7. Connects pelias/pelias#831

BREAKING CHANGE: This drops Elasticsearch 5 from the test matrix for this repo. While it makes no other changes, it's a breaking change as it marks the point where we stop supporting ES5. This allows us to make changes that support only ES6/7 as we add support for Elasticsearch 7. Connects pelias/pelias#831

Now that we have dropped support for ES5, we can change the default type name from `doc` to `_doc`. Either setting is compatible with ES6, but only `_doc` is compatible with ES7. Connects pelias/pelias#831

This drops support for testing the schema with the Elasticsearch type name set to `doc`. This was only needed to support Elasticsearch 5. In order to support Elasticsearch 7, we'll no longer be supporting ES5. Connects pelias/pelias#831

This adds Elasticsearch 7.5.1 to the CI test matrix. Until we merge complete support for ES7, this CI run is allowed to fail without failing the entire build. Connects pelias/pelias#831

In ES7, specifying a mapping type name will no longer be allowed. ES6 can emulate this behavior by setting the `include_type_name` parameter to `false` when creating and fetching mappings. This PR sets that parameter so that our mapping format is compatible with ES6, while using the ES7 preferred format. In the future, when we wish to drop support for ES6, we'll only have to stop using the `include_type_name` configuration option. Connects pelias/pelias#831

The `_all` field was deprecated in Elasticsearch 6 and completely removed in [Elasticsearch 7](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#all-meta-field-removed). Pelias has disabled this field for quite some time, however now that we have dropped support for ES5, we can remove this configuration option. This also moves us towards supporting ES7! Connects pelias/pelias#831

Connects pelias/config#122 Connects pelias/pelias#831

As of `pelias-config-4.8.0` we are now using the new Elasticsearch 7 compatible default document type name: `_doc`. Now that we have dropped support for ES5, we want to ensure this value is the default going forward. Connects pelias/config#122 Connects pelias/pelias#831

Connects pelias/pelias#831

Now that we have merged pelias/config#122 and associated PRs to add Elasticsearch 7 support to Pelias master branches, we no longer need to override the `typeName` config option in project `pelias.json` configration. This reverts commit acc4202. Connects pelias/pelias#831

Connects pelias/pelias#831

orangejulius · 2020-01-09T03:00:49Z

As of today, Elasticsearch 7 support has been merged to the master branches of all relevant Pelias repositories.

We'll follow up with additional changes or improvements as needed, but in our testing so far, ES7 appears to perform well.

We'll also soon start rolling out ES7 as the default to many of the regional projects in the Pelias Docker project.

Connects pelias/pelias#831

This should support at least both ES6 and ES7 counts. Connects pelias/pelias#831

Connects pelias/pelias#831

Its done :) Connects pelias/pelias#831

orangejulius · 2020-01-18T16:53:53Z

I think this is done now! 🎉

In our testing ES7 performs well compared to ES6. There are no changes in query behavior and performance is the same or slightly better.

Please reach out to us if you find any ES7 related bugs! ES6 will continue to be supported for some time, but ES7 is now the recommended version!

mihneadb · 2020-02-21T12:21:25Z

@orangejulius I was trying to figure out if updating to a newer version of Pelias should imply data changes in ES and couldn't find this in any release notes / doc page. Is that a thing? And do you think maybe there should be such a page? I can add it to the docs repo via a PR (just need some tips re: its contents).
Thanks!

orangejulius · 2020-02-21T18:13:25Z

@mihneadb good point.

We definitely recommend starting fresh when updating to a new version, but you can follow the general Elasticsearch compatibility, where in general Elasticsearch can read indices created from one prior major version.

In our experience, using an index created in ES5 with ES6 lead to performance issues, but using an index created in ES6 with ES7 has worked fine.

This has been the case for some time. Connects pelias/pelias#831

orangejulius · 2020-03-13T22:37:54Z

Just in case anyone comes to this issue looking, Elasticsearch 7 is currently not only the highest supported, but also the recommended version of Elasticsearch to use.

Until Elasticsearch 8 comes out and we add support for it, that will continue to be the case.

This is necessary for Elasticsearch 7 Connects pelias/pelias#831

The first error seen when trying to use our current schema with Elasticsearch 7 is: ``` [illegal_argument_exception] Token filter [word_delimiter] cannot be used to parse synonyms ``` The [word delimiter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html) token filter is only used in one place: the `peliasAdmin` analyzer. Looking at the documentation for `word_delimiter`, it does _a lot_: splitting words, handling punctuation, and even some basic stemming. It really feels like an extremely broad tool and at this point feels like something that Elasticsearch would deprecate in the future. Furthermore, looking at our integration tests, it seems one of the key reasons we used it was to tokenize on hyphens, which we have done using the `peliasNameTokenizer` since #375. Considering how complicated this token filter is, and how it's now being used with relatively little effect, it seems like something we can remove. Connects pelias/pelias#831

* feat(elasticsearch): Default to `_doc` as type name for ES7 support Now that we have dropped support for ES5, we can change the default type name from `doc` to `_doc`. Either setting is compatible with ES6, but only `_doc` is compatible with ES7. Connects pelias/pelias#831 * chore(CI): Remove deprecated `matrix` section Connects pelias/pelias#850 * feat(config): Default `whosonfirst.importPostalcodes` to true These take very little additional space, and are quite useful. We should have enabled this a long time ago. Closes pelias#61 * fix(esclient): default esclient.apiVersion to 7.x * feat: remove `imports.whosonfirst.importVenues` * feat(Node.js): Drop support for Node.js 8 Node.js 8 is no longer supported as it reached [end of life](https://github.com/nodejs/Release#release-schedule) at the end of 2019. Connects pelias/pelias#837 * feat: Enable Postal Cities by default For quite a while now we've had a solution to the "Postal Cities problem" (pelias/pelias#396), but it was disabled by default. Enough time has passed that it should probably be enabled. Closes pelias/pelias#396 * fix(get): support for lodash get defaultValue * removed auth * fix syntax Co-authored-by: Julian Simioni <[email protected]> Co-authored-by: Julian Simioni <[email protected]> Co-authored-by: missinglink <[email protected]> Co-authored-by: Joxit <[email protected]>

This is the first step in supporting Elasticsearch 7. At this time, Pelias does not work out of the box on ES7, but with a Docker image ready to go, we can begin testing changes for compatibility. This Dockerfile and config is identical to the ES6 Docker image, except for changing the version, and making one update to the `elasticsearch.yml`: In ES7, the bulk thread pool is removed, and both bulk and non-bulk operations go through a single [write](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool) thread pool. For Pelias we have found increasing the queue size of this thread pool is useful to ensure imports can succeed without errors, so the configuration file has been updated accordingly. Connects pelias/pelias#831

Now that we have merged pelias/config#122 and associated PRs to add Elasticsearch 7 support to Pelias master branches, we no longer need to override the `typeName` config option in project `pelias.json` configration. This reverts commit acc4202. Connects pelias/pelias#831

Connects pelias/pelias#831

orangejulius mentioned this issue Oct 28, 2019

Add Elasticsearch 7.4.0 Docker image pelias/docker#140

Merged

orangejulius added a commit to pelias/model that referenced this issue Nov 7, 2019

feat(ES): Drop type parameter completely

d991450

This is necessary for Elasticsearch 7 Connects pelias/pelias#831

orangejulius mentioned this issue Nov 7, 2019

feat(peliasAdmin): Remove word delimiter filter pelias/schema#392

Closed

missinglink pinned this issue Nov 22, 2019

orangejulius added a commit to pelias/documentation that referenced this issue Jan 3, 2020

Update roadmap from ES6 to ES7

1d8c92f

Elasticsearch 6 is now supported, we are working on ES7! Connects pelias/pelias#719 Connects pelias/pelias#831

orangejulius mentioned this issue Jan 7, 2020

feat(integration): Add total hits helper pelias/schema#417

Merged

orangejulius mentioned this issue Jan 7, 2020

feat(ES5): Drop support for Elasticsearch 5 pelias/schema#418

Merged

orangejulius mentioned this issue Jan 8, 2020

chore(CI): Remove old Elasticsearch type support (doc) from CI pelias/schema#421

Merged

orangejulius mentioned this issue Jan 8, 2020

chore(CI): Add Elasticsearch 7 to CI pelias/schema#422

Merged

This was referenced Jan 8, 2020

feat(ES7): Remove type name from mapping configuration pelias/schema#423

Merged

feat(elasticsearch): Default to _doc as type name for ES7 support pelias/config#122

Merged

orangejulius mentioned this issue Jan 8, 2020

feat(es7): remove _all mapping pelias/schema#424

Merged

orangejulius added a commit to pelias/geonames that referenced this issue Jan 8, 2020

fix(tests): Update test expectations for new doc type name

88803d9

Connects pelias/config#122 Connects pelias/pelias#831

missinglink mentioned this issue Jan 8, 2020

elasticsearch 7 pelias/schema#394

Closed

5 tasks

orangejulius mentioned this issue Jan 8, 2020

Require >= pelias-config-4.8.0 pelias/csv-importer#53

Merged

orangejulius added a commit to pelias/docker that referenced this issue Jan 8, 2020

Use Elasticsearch 7.5.1 across all projects

0cea663

Connects pelias/pelias#831

orangejulius mentioned this issue Jan 8, 2020

Remove typeName configuration option pelias/docker#162

Merged

orangejulius added a commit to pelias/docker that referenced this issue Jan 8, 2020

Use Elasticsearch 7.5.1 across all projects

3714f5d

Connects pelias/pelias#831

orangejulius added a commit to pelias/documentation that referenced this issue Jan 10, 2020

Update list of supported Elasticsearch versions

1e8ee15

Connects pelias/pelias#831

orangejulius added a commit to pelias/dashboard that referenced this issue Jan 15, 2020

feat(count): Get total count from separate call

8f4e401

This should support at least both ES6 and ES7 counts. Connects pelias/pelias#831

This was referenced Jan 16, 2020

Elasticsearch7 pelias/terraform-elasticsearch#16

Merged

Elasticsearch 7 Support pelias/packer-elasticsearch#4

Merged

orangejulius added a commit to pelias/docker that referenced this issue Jan 18, 2020

Use Elasticsearch 7.5.1 across all projects

0ad16cc

Connects pelias/pelias#831

orangejulius mentioned this issue Jan 18, 2020

Use Elasticsearch 7.5.1 across all projects pelias/docker#168

Merged

orangejulius added a commit to pelias/documentation that referenced this issue Jan 18, 2020

Remove ES7 support from roadmap

7d1d68e

Its done :) Connects pelias/pelias#831

orangejulius closed this as completed Jan 18, 2020

orangejulius added a commit to pelias/documentation that referenced this issue Mar 13, 2020

Update to recommending ES7 over ES6

fc2abbc

This has been the case for some time. Connects pelias/pelias#831

orangejulius added a commit to pelias/model that referenced this issue May 20, 2020

feat(ES): Drop type parameter completely

1a31fb6

This is necessary for Elasticsearch 7 Connects pelias/pelias#831

orangejulius unpinned this issue May 26, 2020

calpb pushed a commit to sorelle/docker that referenced this issue Mar 29, 2021

Use Elasticsearch 7.5.1 across all projects

2b3753e

Connects pelias/pelias#831

This was referenced Dec 13, 2021

When to drop support for Elasticsearch 6? #881

Closed

Pelias and log4j zero-day #log4shell CVE-2021-44228 #921

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elasticsearch 7 Support #831

Elasticsearch 7 Support #831

orangejulius commented Oct 28, 2019 •

edited

Loading

orangejulius commented Nov 7, 2019

missinglink commented Nov 26, 2019

missinglink commented Nov 27, 2019 •

edited

Loading

orangejulius commented Jan 9, 2020

orangejulius commented Jan 18, 2020

mihneadb commented Feb 21, 2020

orangejulius commented Feb 21, 2020

orangejulius commented Mar 13, 2020

Elasticsearch 7 Support #831

Elasticsearch 7 Support #831

Comments

orangejulius commented Oct 28, 2019 • edited Loading

Pelias Tasks

Reference links

orangejulius commented Nov 7, 2019

missinglink commented Nov 26, 2019

missinglink commented Nov 27, 2019 • edited Loading

orangejulius commented Jan 9, 2020

orangejulius commented Jan 18, 2020

mihneadb commented Feb 21, 2020

orangejulius commented Feb 21, 2020

orangejulius commented Mar 13, 2020

orangejulius commented Oct 28, 2019 •

edited

Loading

missinglink commented Nov 27, 2019 •

edited

Loading