Skip to content

Conversation

@lhotari
Copy link
Contributor

@lhotari lhotari commented Jun 28, 2023

Motivation

By default, Elasticsearch container will download the geoip lite database.
It's better to disable this behavior by default.
There's also an elastic issue about this default behavior: elastic/elasticsearch#90673. Fixed since v8.7.0 with elastic/elasticsearch#92335 .

Modifications

geoip downloading can be disabled by passing -Dingest.geoip.downloader.enabled.default=false in JVM options or passing ingest.geoip.downloader.enabled=false in the environment.

The problem with passing ingest.geoip.downloader.enabled=false in the environment is that it's not compatible with OpenSearch. OpenSearch will fail to start with a configuration validation error org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [ingest.geoip.downloader.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings. Some users might be using ElasticsearchContainer class together with the OpenSearch docker image.

@lhotari lhotari requested a review from a team June 28, 2023 17:33
@eddumelendez
Copy link
Member

JFTR, OpenSearch provides its own module

@lhotari
Copy link
Contributor Author

lhotari commented Jun 28, 2023

The elastic issue description elastic/elasticsearch#90673 is interesting:

In Elasticsearch 7.14 a feature was added which would automatically download the MaxMind GeoIP database. Specifically, it downloads the database from Google Cloud after fetching this JSON object. This is enabled by default.

We discovered that this feature was responsible for around $20,000 on our AWS bill since we upgraded last March and based on a rough approximation of traffic we were charged for it probably cost Elastic about $5,500 💸 Investigation was slow and required enabling AWS VPC logs to assess the NAT gateway traffic source.

@lhotari
Copy link
Contributor Author

lhotari commented Jun 28, 2023

I guess this geoip download doesn't impact the default image tag that ElasticsearchContainer sets, 7.9.2. The feature was introduced in 7.14.0.
7.14.0 release notes includes:

  • Enable GeoIP downloader by default #74323

@eddumelendez eddumelendez added this to the next milestone Jun 28, 2023
@eddumelendez eddumelendez merged commit b8c4a65 into testcontainers:main Jun 28, 2023
@eddumelendez
Copy link
Member

Thanks again, @lhotari ! The context is really useful, let's keep compatibility in case OpenSearch is used along with ElasticsearchContainer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants