Enable GeoIP downloader by default

docs/reference/ingest/apis/geoip-stats-api.asciidoc

@@ -23,6 +23,9 @@ GET _ingest/geoip/stats
 * If the {es} {security-features} are enabled, you must have the `monitor` or
 `manage` <<privileges-list-cluster,cluster privilege>> to use this API.
 
+* If <<ingest-geoip-downloader-enabled,`ingest.geoip.downloader.enabled`>> is
+disabled, this API returns zero values and an empty `nodes` object.
+
 [role="child_attributes"]
 [[geoip-stats-api-response-body]]
 ==== {api-response-body-title}
@@ -87,4 +90,4 @@ Downloaded database files, including related license files. {es} stores these
 files in the node's <<es-tmpdir,temporary directory>>:
 `$ES_TMPDIR/geoip-databases/<node_id>`.
 =====
-====
+====

docs/reference/ingest/processors/geoip.asciidoc

@@ -4,24 +4,24 @@
 <titleabbrev>GeoIP</titleabbrev>
 ++++
 
-The `geoip` processor adds information about the geographical location of IP addresses, based on data from the Maxmind databases.
-This processor adds this information by default under the `geoip` field. The `geoip` processor can resolve both IPv4 and
-IPv6 addresses.
+The `geoip` processor adds information about the geographical location of an
+IPv4 or IPv6 address.
 
-The `ingest-geoip` module ships by default with the GeoLite2 City, GeoLite2 Country and GeoLite2 ASN GeoIP2 databases from Maxmind made available
-under the CCA-ShareAlike 4.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/
+[[geoip-automatic-updates]]
+By default, the processor uses the GeoLite2 City, GeoLite2 Country, and GeoLite2
+ASN GeoIP2 databases from
+http://dev.maxmind.com/geoip/geoip2/geolite2/[MaxMind], shared under the
+CCA-ShareAlike 4.0 license. {es} automatically downloads updates for
+these databases from the Elastic GeoIP endpoint:
+https://geoip.elastic.co/v1/database. To get download statistics for these
+updates, use the <<geoip-stats-api,GeoIP stats API>>.
 
-The `geoip` processor can run with other city, country and ASN GeoIP2 databases
-from Maxmind. On {ess} deployments, custom database files must be uploaded using
-a {cloud}/ec-custom-bundles.html[custom bundle]. On self-managed deployments,
-custom database files must be copied into the `ingest-geoip` config
-directory located at `$ES_CONFIG/ingest-geoip`.
+If your cluster can't connect to the Elastic GeoIP endpoint or you want to
+manage your own updates, see <<manage-geoip-database-updates>>.
 
-Custom database files must be
-stored uncompressed and the extension must be `-City.mmdb`, `-Country.mmdb`, or
-`-ASN.mmdb` to indicate the type of the database. The
-`database_file` processor option is used to specify the filename of the custom
-database to use for the processor.
+If {es} can't connect to the endpoint for 30 days all updated databases will become
+invalid. {es} will stop enriching documents with geoip data and will add `tags: ["_geoip_expired_database"]`
+field instead.
 
 [[using-ingest-geoip]]
 ==== Using the `geoip` Processor in a Pipeline
@@ -32,7 +32,7 @@ database to use for the processor.
 |======
 | Name                   | Required  | Default                                                                            | Description
 | `field`                | yes       | -                                                                                  | The field to get the ip address from for the geographical lookup.
-| `target_field`         | no        | geoip                                                                              | The field that will hold the geographical information looked up from the Maxmind database.
+| `target_field`         | no        | geoip                                                                              | The field that will hold the geographical information looked up from the MaxMind database.
 | `database_file`        | no        | GeoLite2-City.mmdb                                                                 | The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the `ingest-geoip` config directory.
 | `properties`           | no        | [`continent_name`, `country_iso_code`, `country_name`, `region_iso_code`, `region_name`, `city_name`, `location`] *   | Controls what properties are added to the `target_field` based on the geoip lookup.
 | `ignore_missing`       | no        | `false`                                                                            | If `true` and `field` does not exist, the processor quietly exits without modifying the document
@@ -303,6 +303,82 @@ GET /my_ip_locations/_search
 // TESTRESPONSE[s/"took" : 3/"took" : $body.took/]
 ////
 
+[[manage-geoip-database-updates]]
+==== Manage your own GeoIP2 database updates
+
+If you can't <<geoip-automatic-updates,automatically update>> your GeoIP2
+databases from the Elastic endpoint, you have a few other options:
+
+* <<use-proxy-geoip-endpoint,Use a proxy endpoint>>
+* <<use-custom-geoip-endpoint,Use a custom endpoint>>
+* <<manually-update-geoip-databases,Manually update your GeoIP2 databases>>
+
+[[use-proxy-geoip-endpoint]]
+**Use a proxy endpoint**
+
+If you can't connect directly to the Elastic GeoIP endpoint, consider setting up
+a secure proxy. You can then specify the proxy endpoint URL in the
+<<ingest-geoip-downloader-endpoint,`ingest.geoip.downloader.endpoint`>> setting
+of each node’s `elasticsearch.yml` file.
+
+[[use-custom-geoip-endpoint]]
+**Use a custom endpoint**
+
+You can create a service that mimics the Elastic GeoIP endpoint. You can then
+get automatic updates from this service.
+
+. Download your `.mmdb` database files from the
+http://dev.maxmind.com/geoip/geoip2/geolite2[MaxMind site].
+
+. Copy your database files to a single directory.
+
+. From your {es} directory, run:
++
+[source,sh]
+----
+./bin/elasticsearch-geoip -s my/source/dir [-t target/directory]
+----
+
+. Serve the static database files from your directory. For example, you can use
+Docker to serve the files from an nginx server:
++
+[source,sh]
+----
+docker run -v my/source/dir:/usr/share/nginx/html:ro nginx
+----
+
+. Specify the service's endpoint URL in the
+<<ingest-geoip-downloader-endpoint,`ingest.geoip.downloader.endpoint`>> setting
+of each node’s `elasticsearch.yml` file.
++
+By default, {es} checks the endpoint for updates every three days. To use
+another polling interval, use the <<cluster-update-settings,update cluster
+settings API>> to set
+<<ingest-geoip-downloader-poll-interval,`ingest.geoip.downloader.poll.interval`>>.
+
+[[manually-update-geoip-databases]]
+**Manually update your GeoIP2 databases**
+
+. Use the <<cluster-update-settings,update cluster settings API>> to set
+`ingest.geoip.downloader.enabled` to `false`. This disables automatic updates
+that may overwrite your database changes. This also deletes all downloaded
+databases.
+
+. Download your `.mmdb` database files from the
+http://dev.maxmind.com/geoip/geoip2/geolite2[MaxMind site].
++
+You can also use custom city, country, and ASN `.mmdb` files. These files must
+be uncompressed and use the respective `-City.mmdb`, `-Country.mmdb`, or
+`-ASN.mmdb` extensions.
+
+. On {ess} deployments upload database using
+a {cloud}/ec-custom-bundles.html[custom bundle].
+
+. On self-managed deployments copy the database files to `$ES_CONFIG/ingest-geoip`.
+
+. In your `geoip` processors, configure the `database_file` parameter to use a
+custom database file.
+
 [[ingest-geoip-settings]]
 ===== Node Settings
 
@@ -313,3 +389,28 @@ The `geoip` processor supports the following setting:
     The maximum number of results that should be cached. Defaults to `1000`.
 
 Note that these settings are node settings and apply to all `geoip` processors, i.e. there is one cache for all defined `geoip` processors.
+
+[[geoip-cluster-settings]]
+===== Cluster settings
+
+[[ingest-geoip-downloader-enabled]]
+`ingest.geoip.downloader.enabled`::
+(<<dynamic-cluster-setting,Dynamic>>, Boolean)
+If `true`, {es} automatically downloads and manages updates for GeoIP2 databases
+from the `ingest.geoip.downloader.endpoint`. If `false`, {es} does not download
+updates and deletes all downloaded databases. Defaults to `true`.
+
+[[ingest-geoip-downloader-endpoint]]
+`ingest.geoip.downloader.endpoint`::
+(<<static-cluster-setting,Static>>, string)
+Endpoint URL used to download updates for GeoIP2 databases. Defaults to
+`https://geoip.elastic.co/v1/database`. {es} stores downloaded database files in
+each node's <<es-tmpdir,temporary directory>> at
+`$ES_TMPDIR/geoip-databases/<node_id>`.
+
+[[ingest-geoip-downloader-poll-interval]]
+`ingest.geoip.downloader.poll.interval`::
+(<<dynamic-cluster-setting,Dynamic>>, <<time-units,time value>>)
+How often {es} checks for GeoIP2 database updates at the
+`ingest.geoip.downloader.endpoint`. Must be greater than `1d` (one day). Defaults
+to `3d` (three days).

modules/ingest-geoip/build.gradle

@@ -55,7 +55,6 @@ tasks.named("internalClusterTest").configure {
   if (useFixture) {
     nonInputProperties.systemProperty "geoip_endpoint", "${-> fixtureAddress()}"
   }
-  systemProperty "ingest.geoip.downloader.enabled.default", "true"
 }
 
 tasks.register("copyDefaultGeoIp2DatabaseFiles", Copy) {

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloaderTaskExecutor.java

@@ -37,7 +37,7 @@
  */
 public final class GeoIpDownloaderTaskExecutor extends PersistentTasksExecutor<GeoIpTaskParams> implements ClusterStateListener {
 
-    static final boolean ENABLED_DEFAULT = "true".equals(System.getProperty("ingest.geoip.downloader.enabled.default"));
+    private static final boolean ENABLED_DEFAULT = "false".equals(System.getProperty("ingest.geoip.downloader.enabled.default")) == false;
     public static final Setting<Boolean> ENABLED_SETTING = Setting.boolSetting("ingest.geoip.downloader.enabled", ENABLED_DEFAULT,
         Setting.Property.Dynamic, Setting.Property.NodeScope);
 

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpTaskState.java

@@ -69,7 +69,7 @@ public static GeoIpTaskState fromXContent(XContentParser parser) throws IOExcept
             in -> {
                 long lastUpdate = in.readLong();
                 return new Metadata(lastUpdate, in.readVInt(), in.readVInt(), in.readString(),
-                    in.getVersion().onOrAfter(Version.V_8_0_0) ? in.readLong() : lastUpdate);
+                    in.getVersion().onOrAfter(Version.V_7_14_0) ? in.readLong() : lastUpdate);
             }));
     }
 
@@ -135,7 +135,7 @@ public void writeTo(StreamOutput out) throws IOException {
             o.writeVInt(v.firstChunk);
             o.writeVInt(v.lastChunk);
             o.writeString(v.md5);
-            if (o.getVersion().onOrAfter(Version.V_8_0_0)) {
+            if (o.getVersion().onOrAfter(Version.V_7_14_0)) {
                 o.writeLong(v.lastCheck);
             }
         });

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/IngestGeoIpPlugin.java

@@ -113,10 +113,6 @@ public Collection<Object> createComponents(Client client,
             throw new UncheckedIOException(e);
         }
 
-        if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) {
-            return List.of(databaseRegistry.get());
-        }
-
         geoIpDownloaderTaskExecutor = new GeoIpDownloaderTaskExecutor(client, new HttpClient(), clusterService, threadPool);
         return List.of(databaseRegistry.get(), geoIpDownloaderTaskExecutor);
     }
@@ -130,9 +126,6 @@ public void close() throws IOException {
     public List<PersistentTasksExecutor<?>> getPersistentTasksExecutor(ClusterService clusterService, ThreadPool threadPool,
                                                                        Client client, SettingsModule settingsModule,
                                                                        IndexNameExpressionResolver expressionResolver) {
-        if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) {
-            return Collections.emptyList();
-        }
         return List.of(geoIpDownloaderTaskExecutor);
     }
 
@@ -165,9 +158,6 @@ public List<NamedWriteableRegistry.Entry> getNamedWriteables() {
 
     @Override
     public Collection<SystemIndexDescriptor> getSystemIndexDescriptors(Settings settings) {
-        if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) {
-            return Collections.emptyList();
-        }
         SystemIndexDescriptor geoipDatabasesIndex = SystemIndexDescriptor.builder()
             .setIndexPattern(DATABASES_INDEX)
             .setDescription("GeoIP databases")

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStats.java

@@ -59,7 +59,7 @@ public GeoIpDownloaderStats(StreamInput in) throws IOException {
         totalDownloadTime = in.readVLong();
         databasesCount = in.readVInt();
         skippedDownloads = in.readVInt();
-        if (in.getVersion().onOrAfter(Version.V_8_0_0)) {
+        if (in.getVersion().onOrAfter(Version.V_7_14_0)) {
             expiredDatabases = in.readVInt();
         } else {
             expiredDatabases = 0;
@@ -149,7 +149,7 @@ public void writeTo(StreamOutput out) throws IOException {
         out.writeVLong(totalDownloadTime);
         out.writeVInt(databasesCount);
         out.writeVInt(skippedDownloads);
-        if (out.getVersion().onOrAfter(Version.V_8_0_0)) {
+        if (out.getVersion().onOrAfter(Version.V_7_14_0)) {
             out.writeVInt(expiredDatabases);
         }
     }

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStatsTransportAction.java

@@ -33,20 +33,17 @@ public class GeoIpDownloaderStatsTransportAction extends TransportNodesAction<Re
 
     private final TransportService transportService;
     private final DatabaseRegistry registry;
-    private GeoIpDownloaderTaskExecutor geoIpDownloaderTaskExecutor;
+    private final GeoIpDownloaderTaskExecutor geoIpDownloaderTaskExecutor;
 
     @Inject
     public GeoIpDownloaderStatsTransportAction(TransportService transportService, ClusterService clusterService,
-                                               ThreadPool threadPool, ActionFilters actionFilters, DatabaseRegistry registry) {
+                                               ThreadPool threadPool, ActionFilters actionFilters, DatabaseRegistry registry,
+                                               GeoIpDownloaderTaskExecutor geoIpDownloaderTaskExecutor) {
         super(GeoIpDownloaderStatsAction.NAME, threadPool, clusterService, transportService, actionFilters, Request::new,
             NodeRequest::new, ThreadPool.Names.MANAGEMENT, NodeResponse.class);
         this.transportService = transportService;
         this.registry = registry;
-    }
-
-    @Inject(optional = true)
-    public void setTaskExecutor(GeoIpDownloaderTaskExecutor taskExecutor){
-        geoIpDownloaderTaskExecutor = taskExecutor;
+        this.geoIpDownloaderTaskExecutor = geoIpDownloaderTaskExecutor;
     }
 
     @Override
@@ -66,7 +63,7 @@ protected NodeResponse newNodeResponse(StreamInput in) throws IOException {
 
     @Override
     protected NodeResponse nodeOperation(NodeRequest request, Task task) {
-        GeoIpDownloader geoIpTask = geoIpDownloaderTaskExecutor == null ? null : geoIpDownloaderTaskExecutor.getCurrentTask();
+        GeoIpDownloader geoIpTask = geoIpDownloaderTaskExecutor.getCurrentTask();
         GeoIpDownloaderStats stats = geoIpTask == null || geoIpTask.getStatus() == null ? null : geoIpTask.getStatus();
         return new NodeResponse(transportService.getLocalNode(), stats, registry.getAvailableDatabases(), registry.getFilesInTemp());
     }