Fix for ES7 : Specifying types is deprecated.

[oxarbitrage]

Note: this PR contains breaking changes.

With this PR, the node attempts to obtain the version of ES at startup, and if it fails (for example, due to lack of permissions), the ES server is assumed to be version 7 or higher.

Before this change, in ES, we use the path pattern bitshares-*/data/2.9.x, where data is the type. After this change, we stop specifying the type if ES is version 7 or higher, and the path pattern becomes bitshares-*/2.9.x for newly auto-created indexes (Note: bitshares-*/_doc/2.9.x is usable with ES7 but not ES8, see #2707 for more info). Of course, the existing data will not change by itself. The same applies to the objects-* indexes.

• ES service maintainers need to manually delete all existing indexes from their ES databases and regenerate all data with the new code, or use a new prefix other than the default one (bitshares-), or install a new ES database. This may take days or weeks to complete. If the old data is not deleted, it uses around 500G of disk space.
  By the way, we need more disk space to store other new data due to Store some data in ES as objects instead of just as strings #2565 and etc.
• Client libraries and applications need to query data through the new path pattern and, if applicable, a new prefix.

Those who are interested, please check #2240 (comment) for more info.

---

The original OP is below.

Reported in telegram by @HarukaMa . Elasticsearch plugin is not working in ES7 as _type is deprecated. However this needs to be there for ES6(and probably lower versions).

This pull send a query to the server, get the version string and send the type accordingly in both plugins(elasticsearch and es_objects)

[abitmore]

I'm not sure whether this PR fixed a problem, but my node is running ES7 fine with default config.ini.

My ES node was installed following this guide: https://www.elastic.co/guide/en/elasticsearch/reference/7.3/deb.html

The issue @HarukaMa reported was an error when elasticsearch-mode parameter is set to 2 (perhaps not related to this PR):

# Mode of operation: only_save(0), only_query(1), all(2) - Default: 0
# elasticsearch-mode = 

[oxarbitrage]

He later reported the issue was happening for him also in only_save(0) iirc.

[oxarbitrage]

In conversations in the dev channel yesterday he fixed by removing the _type from the queries just as we are doing here except that we need to keep functionality for ES6. I am positive that it will not work with types removed on ES6.
Not sure why your ES7 is working without this change while his not. A different minor version ?

Will try ES7 myself.

[oxarbitrage]

I found the details of the error @HarukaMa was getting:

Rejecting mapping update to [bitshares-2015-10] as the final mapping would have more than 1 type: [_doc, data]

Version string is: 7.3.1

@abitmore can you post your version? curl -XGET http://localhost:9200/ will do it.

[abitmore]

My node is 7.3.0.

[HarukaMa]

Now thinking that, I think it is possible that is because I have templates to adjust the parameters (like suppressing refreshing during initial index, changing replica counts and some settings optimized for spinning disks), and the template itself may have implied _doc type although it’s not explicitly set. Anyways, the type field itself is indeed deprecated. iPhoneから送信 2019/09/22 8:32、Abit <[email protected]>のメール:

…

My node is 7.3.0. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[oxarbitrage]

This by now is not needed. i downloaded and tested elasticsearch 7.4.0 (last stable version) without any issue. The pull can be handy in the future but not needed now.

[abitmore]

This is still needed. In ES8 the warning will become an error.

[abitmore]

Those who are interested, please check #2240 (comment) for more info.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

67.6% Coverage
0.0% Duplication