search_phase_execution_exception

[fredliporace]

Search endpoint is currently raising the following exception:

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.'

[fredliporace]

First occurrence at 2021-10-15T08:36:21.380-03:00. No stack update event in the past months.

[fredliporace]

The es index and all its documents were dropped. I suspect a node failure was the cause. The index will have to be recreated and all the documents indexed again.

[felippehsk]

First of all, thank you so much for putting this STAC and keeping it running, @fredliporace.

Do we have any updates in this issue? The error still keeps showing...

[fredliporace]

hi @felippehsk , hope to have it fixed by the end of this month. I'll post updates here and on cbers-pds OSGEO list.

[fredliporace]

While I'm still working on a procedure to recover from ES cluster fails there is a temporary endpoint that contains the scenes ingested last week and will be updated with new scenes. I'll keep this online until the original API is reestablished with the full archive.

https://stac.amskepler.com/partial

[fredliporace]

Service is back online

[felippehsk]

@fredliporace , thank you so much for working on this and putting the service back on! Cheers!

[fredliporace]

Noticed now that the same error is happening again at the live version. The stac index mappings seems to be one automatically created, although the auto creation should be disabled exactly for these kind of problems to be reported by DLQs messages as the index is corrupted.

The mapping for geometry for example should be geo_shape but is currently the default text.

Although the software may be improved to detect this situation sooner and allow a proper index recovery from the backup, the reason this situation pops up now and then in the live service is its es cluster configuration - I'm using a single node t2.small.search, which is clearly inadequate for a production service.

Unfortunately the Opensearch service is the main drive of cost in operating the live version and I don't have plans now to upgrade to a suitable configuration for production service. I'll then keep the live version limited to support testing, with only a subset of the AWS archive, which may be reset from time to time if such index corruption happens again. Organizations or individuals interested in running a full production service should be able to do so using new deployments.

Note that this only affects the STAC API search, the static STAC index in the live version reflects the complete AWS archive.

[fredliporace]

Service now covers all archive, thanks to AWS support.