diff --git a/docs/reference/ingest/apis/geoip-stats-api.asciidoc b/docs/reference/ingest/apis/geoip-stats-api.asciidoc index 6ef0db546342b..dda1f1a2d1492 100644 --- a/docs/reference/ingest/apis/geoip-stats-api.asciidoc +++ b/docs/reference/ingest/apis/geoip-stats-api.asciidoc @@ -23,9 +23,6 @@ GET _ingest/geoip/stats * If the {es} {security-features} are enabled, you must have the `monitor` or `manage` <> to use this API. -* If <> is -disabled, this API returns zero values and an empty `nodes` object. - [role="child_attributes"] [[geoip-stats-api-response-body]] ==== {api-response-body-title} @@ -90,4 +87,4 @@ Downloaded database files, including related license files. {es} stores these files in the node's <>: `$ES_TMPDIR/geoip-databases/`. ===== -==== \ No newline at end of file +==== diff --git a/docs/reference/ingest/processors/geoip.asciidoc b/docs/reference/ingest/processors/geoip.asciidoc index bde43310fc046..e562d0faf0037 100644 --- a/docs/reference/ingest/processors/geoip.asciidoc +++ b/docs/reference/ingest/processors/geoip.asciidoc @@ -4,20 +4,20 @@ GeoIP ++++ -The `geoip` processor adds information about the geographical location of an -IPv4 or IPv6 address. +The `geoip` processor adds information about the geographical location of IP addresses, based on data from the Maxmind databases. +This processor adds this information by default under the `geoip` field. The `geoip` processor can resolve both IPv4 and +IPv6 addresses. -[[geoip-automatic-updates]] -By default, the processor uses the GeoLite2 City, GeoLite2 Country, and GeoLite2 -ASN GeoIP2 databases from -http://dev.maxmind.com/geoip/geoip2/geolite2/[MaxMind], shared under the -CCA-ShareAlike 4.0 license. {es} automatically downloads updates for -these databases from the Elastic GeoIP endpoint: -https://geoip.elastic.co/v1/database. To get download statistics for these -updates, use the <>. +The `ingest-geoip` module ships by default with the GeoLite2 City, GeoLite2 Country and GeoLite2 ASN GeoIP2 databases from Maxmind made available +under the CCA-ShareAlike 4.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/ -If your cluster can't connect to the Elastic GeoIP endpoint or you want to -manage your own updates, see <>. +The `geoip` processor can run with other city, country and ASN GeoIP2 databases +from Maxmind. The database files must be copied into the `ingest-geoip` config +directory located at `$ES_CONFIG/ingest-geoip`. Custom database files must be +stored uncompressed and the extension must be `-City.mmdb`, `-Country.mmdb`, or +`-ASN.mmdb` to indicate the type of the database. The +`database_file` processor option is used to specify the filename of the custom +database to use for the processor. [[using-ingest-geoip]] ==== Using the `geoip` Processor in a Pipeline @@ -28,7 +28,7 @@ manage your own updates, see <>. |====== | Name | Required | Default | Description | `field` | yes | - | The field to get the ip address from for the geographical lookup. -| `target_field` | no | geoip | The field that will hold the geographical information looked up from the MaxMind database. +| `target_field` | no | geoip | The field that will hold the geographical information looked up from the Maxmind database. | `database_file` | no | GeoLite2-City.mmdb | The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the `ingest-geoip` config directory. | `properties` | no | [`continent_name`, `country_iso_code`, `country_name`, `region_iso_code`, `region_name`, `city_name`, `location`] * | Controls what properties are added to the `target_field` based on the geoip lookup. | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document @@ -303,79 +303,6 @@ GET /my_ip_locations/_search // TESTRESPONSE[s/"took" : 3/"took" : $body.took/] //// -[[manage-geoip-database-updates]] -==== Manage your own GeoIP2 database updates - -If you can't <> your GeoIP2 -databases from the Elastic endpoint, you have a few other options: - -* <> -* <> -* <> - -[[use-proxy-geoip-endpoint]] -**Use a proxy endpoint** - -If you can't connect directly to the Elastic GeoIP endpoint, consider setting up -a secure proxy. You can then specify the proxy endpoint URL in the -<> setting -of each node’s `elasticsearch.yml` file. - -[[use-custom-geoip-endpoint]] -**Use a custom endpoint** - -You can create a service that mimics the Elastic GeoIP endpoint. You can then -get automatic updates from this service. - -. Download your `.mmdb` database files from the -http://dev.maxmind.com/geoip/geoip2/geolite2[MaxMind site]. - -. Copy your database files to a single directory. - -. From your {es} directory, run: -+ -[source,sh] ----- -./bin/elasticsearch-geoip -s my/source/dir [-t target/directory] ----- - -. Serve the static database files from your directory. For example, you can use -Docker to serve the files from an nginx server: -+ -[source,sh] ----- -docker run -v my/source/dir:/usr/share/nginx/html:ro nginx ----- - -. Specify the service's endpoint URL in the -<> setting -of each node’s `elasticsearch.yml` file. -+ -By default, {es} checks the endpoint for updates every three days. To use -another polling interval, use the <> to set -<>. - -[[manually-update-geoip-databases]] -**Manually update your GeoIP2 databases** - -. Use the <> to set -`ingest.geoip.downloader.enabled` to `false`. This disables automatic updates -that may overwrite your database changes. This also deletes all downloaded -databases. - -. Download your `.mmdb` database files from the -http://dev.maxmind.com/geoip/geoip2/geolite2[MaxMind site]. -+ -You can also use custom city, country, and ASN `.mmdb` files. These files must -be uncompressed and use the respective `-City.mmdb`, `-Country.mmdb`, or -`-ASN.mmdb` extensions. - -. Copy the database files to `$ES_CONFIG/ingest-geoip`. - -. In your `geoip` processors, configure the `database_file` parameter to use a -custom database file. - [[ingest-geoip-settings]] ===== Node Settings @@ -386,28 +313,3 @@ The `geoip` processor supports the following setting: The maximum number of results that should be cached. Defaults to `1000`. Note that these settings are node settings and apply to all `geoip` processors, i.e. there is one cache for all defined `geoip` processors. - -[[geoip-cluster-settings]] -===== Cluster settings - -[[ingest-geoip-downloader-enabled]] -`ingest.geoip.downloader.enabled`:: -(<>, Boolean) -If `true`, {es} automatically downloads and manages updates for GeoIP2 databases -from the `ingest.geoip.downloader.endpoint`. If `false`, {es} does not download -updates and deletes all downloaded databases. Defaults to `true`. - -[[ingest-geoip-downloader-endpoint]] -`ingest.geoip.downloader.endpoint`:: -(<>, string) -Endpoint URL used to download updates for GeoIP2 databases. Defaults to -`https://geoip.elastic.co/v1/database`. {es} stores downloaded database files in -each node's <> at -`$ES_TMPDIR/geoip-databases/`. - -[[ingest-geoip-downloader-poll-interval]] -`ingest.geoip.downloader.poll.interval`:: -(<>, <>) -How often {es} checks for GeoIP2 database updates at the -`ingest.geoip.downloader.endpoint`. Must be greater than `1d` (one day). Defaults -to `3d` (three days). diff --git a/modules/ingest-geoip/build.gradle b/modules/ingest-geoip/build.gradle index eced11cf01253..3ec5bea419d5b 100644 --- a/modules/ingest-geoip/build.gradle +++ b/modules/ingest-geoip/build.gradle @@ -54,6 +54,7 @@ tasks.named("internalClusterTest").configure { if (useFixture) { nonInputProperties.systemProperty "geoip_endpoint", "${-> fixtureAddress()}" } + systemProperty "ingest.geoip.downloader.enabled.default", "true" } tasks.register("copyDefaultGeoIp2DatabaseFiles", Copy) { diff --git a/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloaderTaskExecutor.java b/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloaderTaskExecutor.java index a2dc5804f566c..c6a1c3a87e606 100644 --- a/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloaderTaskExecutor.java +++ b/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloaderTaskExecutor.java @@ -37,7 +37,7 @@ */ public final class GeoIpDownloaderTaskExecutor extends PersistentTasksExecutor implements ClusterStateListener { - private static final boolean ENABLED_DEFAULT = "false".equals(System.getProperty("ingest.geoip.downloader.enabled.default")) == false; + static final boolean ENABLED_DEFAULT = "true".equals(System.getProperty("ingest.geoip.downloader.enabled.default")); public static final Setting ENABLED_SETTING = Setting.boolSetting("ingest.geoip.downloader.enabled", ENABLED_DEFAULT, Setting.Property.Dynamic, Setting.Property.NodeScope); diff --git a/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/IngestGeoIpPlugin.java b/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/IngestGeoIpPlugin.java index 92b33475dd5c2..5e076e98e5d7f 100644 --- a/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/IngestGeoIpPlugin.java +++ b/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/IngestGeoIpPlugin.java @@ -113,6 +113,10 @@ public Collection createComponents(Client client, throw new UncheckedIOException(e); } + if(GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false){ + return Collections.singletonList(databaseRegistry.get()); + } + geoIpDownloaderTaskExecutor = new GeoIpDownloaderTaskExecutor(client, new HttpClient(), clusterService, threadPool); return Arrays.asList(databaseRegistry.get(), geoIpDownloaderTaskExecutor); } @@ -126,6 +130,9 @@ public void close() throws IOException { public List> getPersistentTasksExecutor(ClusterService clusterService, ThreadPool threadPool, Client client, SettingsModule settingsModule, IndexNameExpressionResolver expressionResolver) { + if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) { + return Collections.emptyList(); + } return Collections.singletonList(geoIpDownloaderTaskExecutor); } @@ -159,6 +166,9 @@ public List getNamedWriteables() { @Override public Collection getSystemIndexDescriptors(Settings settings) { + if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) { + return Collections.emptyList(); + } SystemIndexDescriptor geoipDatabasesIndex = SystemIndexDescriptor.builder() .setIndexPattern(DATABASES_INDEX) .setDescription("GeoIP databases") @@ -167,7 +177,6 @@ public Collection getSystemIndexDescriptors(Settings sett .put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1) .put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, 0) .put(IndexMetadata.SETTING_AUTO_EXPAND_REPLICAS, "0-1") - .put(IndexMetadata.SETTING_INDEX_HIDDEN, true) .build()) .setOrigin("geoip") .setVersionMetaKey("version") diff --git a/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStatsTransportAction.java b/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStatsTransportAction.java index 38e9c7e8c437d..d177577de73bc 100644 --- a/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStatsTransportAction.java +++ b/modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStatsTransportAction.java @@ -31,17 +31,20 @@ public class GeoIpDownloaderStatsTransportAction extends TransportNodesAction