Add setting to ignore dynamic fields when field limit is reached

docs/changelog/96235.yaml

@@ -0,0 +1,5 @@
+pr: 96235
+summary: Add `true_until_limit` option to `dynamic` mapping parameter to ignore dynamic fields when field limit is reached
+area: Mapping
+type: enhancement
+issues: []

docs/reference/mapping/fields/ignored-field.asciidoc

@@ -4,8 +4,10 @@
 The `_ignored` field indexes and stores the names of every field in a document
 that has been ignored when the document was indexed. This can, for example,
 be the case when the field was malformed and <<ignore-malformed,`ignore_malformed`>>
-was turned on, or when a `keyword` fields value exceeds its optional
-<<ignore-above,`ignore_above`>> setting.
+was turned on, when a `keyword` field's value exceeds its optional
+<<ignore-above,`ignore_above`>> setting, or when
+<<mapping-settings-limit,`index.mapping.total_fields.limit`>> has been reached and
+the mapping parameter <<dynamic-parameters, `dynamic: true_until_limit`>> is set.
 
 This field is searchable with <<query-dsl-term-query,`term`>>,
 <<query-dsl-terms-query,`terms`>> and <<query-dsl-exists-query,`exists`>>

docs/reference/mapping/mapping-settings-limit.asciidoc

@@ -20,7 +20,8 @@ limits the maximum number of clauses in a query.
 +
 [TIP]
 ====
-If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type.
+If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type,
+or setting the mapping parameter <<dynamic-parameters, `dynamic: true_until_limit`>>.
 ====
 
 `index.mapping.depth.limit`::

docs/reference/mapping/params/dynamic.asciidoc

@@ -83,10 +83,18 @@ accepts the following parameters:
 
 [horizontal]
 `true`::    New fields are added to the mapping (default).
+            If the field limit (<<mapping-settings-limit,`index.mapping.total_fields.limit`>>) is exceeded as a result of adding a dynamic
+            field to the mapping, the corresponding index request fails with the message `Limit of total fields [X] has been exceeded`.
 `runtime`:: New fields are added to the mapping as <<runtime,runtime fields>>.
             These fields are not indexed, and are loaded from `_source` at query time.
+            If the field limit (<<mapping-settings-limit,`index.mapping.total_fields.limit`>>) is exceeded as a result of adding a dynamic
+            field to the mapping, the corresponding index request fails with the message `Limit of total fields [X] has been exceeded`.
 `false`::   New fields are ignored. These fields will not be indexed
             or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added
             to the mapping, and new fields must be added explicitly.
 `strict`::  If new fields are detected, an exception is thrown and the document
             is rejected. New fields must be explicitly added to the mapping.
+`true_until_limit`:: New fields are added to the mapping as long as the field limit
+                (<<mapping-settings-limit,`index.mapping.total_fields.limit`>>) is not exceeded.
+                Fields that would exceed the limit are not added to the mapping, similar to `dynamic: false`.
+                Ignored fields are added to the <<mapping-ignored-field, `_ignored` metadata field>>.

docs/reference/troubleshooting/common-issues/mapping-explosion.asciidoc

@@ -41,20 +41,24 @@ timing out in the browser's Developer Tools Network tab.
 doesn't normally cause problems unless it's combined with overriding
 <<mapping-settings-limit,`index.mapping.total_fields.limit`>>. The
 default `1000` limit is considered generous, though overriding to `10000`
-doesn't cause noticable impact depending on use case. However, to give
+doesn't cause noticeable impact depending on use case. However, to give
 a bad example, overriding to `100000` and this limit being hit
 by mapping totals would usually have strong performance implications.
 
 If your index mapped fields expect to contain a large, arbitrary set of
 keys, you may instead consider:
 
+* Setting the mapping parameter <<dynamic-parameters, `dynamic: true_until_limit`>>.
+Instead of rejecting documents that add dynamic fields when the field limit is reached, this will ignore dynamic fields beyond the limit.
+
 * Using the <<flattened,flattened>> data type. Please note,
 however, that flattened objects is link:https://github.com/elastic/kibana/issues/25820[not fully supported in {kib}] yet. For example, this could apply to sub-mappings like { `host.name` ,
 `host.os`, `host.version` }. Desired fields are still accessed by
 <<runtime-search-request,runtime fields>>.
 
 * Disable <<dynamic-mapping,dynamic mappings>>.
-This cannot effect current index mapping, but can apply going forward via an <<index-templates,index template>>.
+ This cannot effect current index mapping, but can apply
+going forward via an <<index-templates,index template>>.
 
 Modifying to the <<nested,nested>> data type would not resolve the core
 issue.

modules/data-streams/src/test/java/org/elasticsearch/datastreams/DataStreamGetWriteIndexTests.java

@@ -239,7 +239,7 @@ public void setup() throws Exception {
                 new MetadataFieldMapper[] { dtfm },
                 Collections.emptyMap()
             );
-            MappingLookup mappingLookup = MappingLookup.fromMappers(mapping, List.of(dtfm, dateFieldMapper), List.of(), List.of());
+            MappingLookup mappingLookup = MappingLookup.fromMappers(mapping, List.of(dtfm, dateFieldMapper), List.of(), List.of(), 1);
             indicesService = DataStreamTestHelper.mockIndicesServices(mappingLookup);
         }
 

modules/legacy-geo/src/test/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapperTests.java

@@ -103,6 +103,18 @@ protected void registerParameters(ParameterChecker checker) throws IOException {
         checker.registerUpdateCheck(b -> b.field("distance_error_pct", 0.8), m -> {});
     }
 
+    @Override
+    public void testMapperBuilderSize() throws IOException {
+        super.testMapperBuilderSize();
+        assertWarnings("Parameter [strategy] is deprecated and will be removed in a future version");
+    }
+
+    @Override
+    public void testMapperBuilderSizeMultiField() throws IOException {
+        super.testMapperBuilderSizeMultiField();
+        assertWarnings("Parameter [strategy] is deprecated and will be removed in a future version");
+    }
+
     @Override
     protected Collection<? extends Plugin> getPlugins() {
         return List.of(new TestLegacyGeoShapeFieldMapperPlugin());

server/src/internalClusterTest/java/org/elasticsearch/index/mapper/DynamicMappingIT.java

@@ -27,9 +27,11 @@
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.core.TimeValue;
 import org.elasticsearch.index.query.GeoBoundingBoxQueryBuilder;
+import org.elasticsearch.index.query.MatchAllQueryBuilder;
 import org.elasticsearch.index.query.MatchQueryBuilder;
 import org.elasticsearch.plugins.Plugin;
 import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.search.SearchHit;
 import org.elasticsearch.test.ESIntegTestCase;
 import org.elasticsearch.test.InternalSettingsPlugin;
 import org.elasticsearch.xcontent.XContentBuilder;
@@ -39,10 +41,13 @@
 
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.atomic.AtomicReference;
 
@@ -51,8 +56,10 @@
 import static org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.MIN_DIMS_FOR_DYNAMIC_FLOAT_MAPPING;
 import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
 import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertSearchHits;
+import static org.hamcrest.Matchers.aMapWithSize;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasKey;
 import static org.hamcrest.Matchers.instanceOf;
 
 public class DynamicMappingIT extends ESIntegTestCase {
@@ -90,18 +97,60 @@ public void testConflictingDynamicMappingsBulk() {
         assertTrue(bulkResponse.hasFailures());
     }
 
-    private static void assertMappingsHaveField(GetMappingsResponse mappings, String index, String field) throws IOException {
-        MappingMetadata indexMappings = mappings.getMappings().get("index");
-        assertNotNull(indexMappings);
-        Map<String, Object> typeMappingsMap = indexMappings.getSourceAsMap();
-        @SuppressWarnings("unchecked")
-        Map<String, Object> properties = (Map<String, Object>) typeMappingsMap.get("properties");
-        assertTrue("Could not find [" + field + "] in " + typeMappingsMap.toString(), properties.containsKey(field));
+    public void testArrayWithDifferentTypes() {
+        createIndex("index");
+        BulkResponse bulkResponse = client().prepareBulk()
+            .add(client().prepareIndex("index").setId("1").setSource("foo", List.of(42, "bar")))
+            .get();
+
+        assertTrue(bulkResponse.hasFailures());
+        assertEquals(
+            "mapper [foo] cannot be changed from type [long] to [text]",
+            bulkResponse.getItems()[0].getFailure().getCause().getMessage()
+        );
+    }
+
+    public void testArraysCountAsOneTowardsFieldLimit() {
+        createIndex("index", Settings.builder().put(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING.getKey(), 2).build());
+        BulkResponse bulkResponse = client().prepareBulk()
+            .add(client().prepareIndex("index").setId("1").setSource("field1", List.of(1, 2), "field2", 1))
+            .get();
+
+        assertFalse(bulkResponse.hasFailures());
     }
 
     public void testConcurrentDynamicUpdates() throws Throwable {
-        createIndex("index");
-        final Thread[] indexThreads = new Thread[32];
+        int numberOfFieldsToCreate = 32;
+        Map<String, Object> properties = indexConcurrently(numberOfFieldsToCreate, Settings.builder(), Map.of());
+        assertThat(properties, aMapWithSize(numberOfFieldsToCreate));
+        for (int i = 0; i < numberOfFieldsToCreate; i++) {
+            assertThat(properties, hasKey("field" + i));
+        }
+    }
+
+    public void testConcurrentDynamicIgnoreBeyondLimitUpdates() throws Throwable {
+        int numberOfFieldsToCreate = 32;
+        Map<String, Object> properties = indexConcurrently(
+            numberOfFieldsToCreate,
+            Settings.builder().put(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING.getKey(), numberOfFieldsToCreate),
+            Map.of("dynamic", "true_until_limit")
+        );
+        // every field is a multi-field (text + keyword)
+        assertThat(properties, aMapWithSize(16));
+        SearchResponse response = client().prepareSearch("index")
+            .setQuery(new MatchAllQueryBuilder())
+            .setSize(numberOfFieldsToCreate)
+            .addFetchField("*")
+            .get();
+        long ignoredFields = Arrays.stream(response.getHits().getHits()).filter(hit -> hit.field("_ignored") != null).count();
+        assertEquals(16, ignoredFields);
+    }
+
+    private Map<String, Object> indexConcurrently(int numberOfFieldsToCreate, Settings.Builder settings, Map<String, Object> mapping)
+        throws Throwable {
+        indicesAdmin().prepareCreate("index").setSettings(settings).setMapping(mapping).get();
+        ensureGreen("index");
+        final Thread[] indexThreads = new Thread[numberOfFieldsToCreate];
         final CountDownLatch startLatch = new CountDownLatch(1);
         final AtomicReference<Throwable> error = new AtomicReference<>();
         for (int i = 0; i < indexThreads.length; ++i) {
@@ -129,14 +178,17 @@ public void run() {
         if (error.get() != null) {
             throw error.get();
         }
-        Thread.sleep(2000);
-        GetMappingsResponse mappings = indicesAdmin().prepareGetMappings("index").get();
-        for (int i = 0; i < indexThreads.length; ++i) {
-            assertMappingsHaveField(mappings, "index", "field" + i);
-        }
-        for (int i = 0; i < indexThreads.length; ++i) {
+        client().admin().indices().prepareRefresh("index").get();
+        for (int i = 0; i < numberOfFieldsToCreate; ++i) {
             assertTrue(client().prepareGet("index", Integer.toString(i)).get().isExists());
         }
+        GetMappingsResponse mappings = indicesAdmin().prepareGetMappings("index").get();
+        MappingMetadata indexMappings = mappings.getMappings().get("index");
+        assertNotNull(indexMappings);
+        Map<String, Object> typeMappingsMap = indexMappings.getSourceAsMap();
+        @SuppressWarnings("unchecked")
+        Map<String, Object> properties = (Map<String, Object>) typeMappingsMap.get("properties");
+        return properties;
     }
 
     public void testPreflightCheckAvoidsMaster() throws InterruptedException, IOException {
@@ -227,15 +279,111 @@ public void onFailure(Exception e) {
             Exception e = expectThrows(DocumentParsingException.class, () -> indexRequestBuilder.get(TimeValue.timeValueSeconds(10)));
             assertThat(e.getMessage(), Matchers.containsString("failed to parse"));
             assertThat(e.getCause(), instanceOf(IllegalArgumentException.class));
-            assertThat(
-                e.getCause().getMessage(),
-                Matchers.containsString("Limit of total fields [2] has been exceeded while adding new fields [1]")
-            );
+            assertThat(e.getCause().getMessage(), Matchers.containsString("Limit of total fields [2] has been exceeded"));
         } finally {
             indexingCompletedLatch.countDown();
         }
     }
 
+    public void testIgnoreDynamicBeyondLimitMultiField() throws Exception {
+        var fields = indexIgnoreDynamicBeyond(2, orderedMap("field1", 1, "field2", "text")).getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("field1", "_ignored")));
+        assertThat(fields.get("field1").getValues(), equalTo(List.of(1L)));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("field2")));
+    }
+
+    public void testIgnoreDynamicArrayField() throws Exception {
+        var fields = indexIgnoreDynamicBeyond(1, orderedMap("field1", 1, "field2", List.of(1, 2))).getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("field1", "_ignored")));
+        assertThat(fields.get("field1").getValues(), equalTo(List.of(1L)));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("field2")));
+    }
+
+    public void testIgnoreDynamicBeyondLimitObjectField() throws Exception {
+        var fields = indexIgnoreDynamicBeyond(3, orderedMap("a.b", 1, "a.c", 2, "a.d", 3)).getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("a.b", "a.c", "_ignored")));
+        assertThat(fields.get("a.b").getValues(), equalTo(List.of(1L)));
+        assertThat(fields.get("a.c").getValues(), equalTo(List.of(2L)));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("a.d")));
+    }
+
+    public void testIgnoreDynamicBeyondLimitObjectField2() throws Exception {
+        var fields = indexIgnoreDynamicBeyond(3, orderedMap("a.b", 1, "a.c", 2, "b.a", 3)).getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("a.b", "a.c", "_ignored")));
+        assertThat(fields.get("a.b").getValues(), equalTo(List.of(1L)));
+        assertThat(fields.get("a.c").getValues(), equalTo(List.of(2L)));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("b")));
+    }
+
+    public void testIgnoreDynamicBeyondLimitDottedObjectMultiField() throws Exception {
+        var fields = indexIgnoreDynamicBeyond(4, orderedMap("a.b", "foo", "a.c", 2, "a.d", 3)).getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("a.b", "a.b.keyword", "a.c", "_ignored")));
+        assertThat(fields.get("a.b").getValues(), equalTo(List.of("foo")));
+        assertThat(fields.get("a.b.keyword").getValues(), equalTo(List.of("foo")));
+        assertThat(fields.get("a.c").getValues(), equalTo(List.of(2L)));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("a.d")));
+    }
+
+    public void testIgnoreDynamicBeyondLimitObjectMultiField() throws Exception {
+        var fields = indexIgnoreDynamicBeyond(5, orderedMap("a", orderedMap("b", "foo", "c", "bar", "d", 3))).getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("a.b", "a.b.keyword", "a.c", "a.c.keyword", "_ignored")));
+        assertThat(fields.get("a.b").getValues(), equalTo(List.of("foo")));
+        assertThat(fields.get("a.b.keyword").getValues(), equalTo(List.of("foo")));
+        assertThat(fields.get("a.c").getValues(), equalTo(List.of("bar")));
+        assertThat(fields.get("a.c.keyword").getValues(), equalTo(List.of("bar")));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("a.d")));
+    }
+
+    public void testFieldLimitRuntimeAndDynamic() throws Exception {
+        assertAcked(
+            indicesAdmin().prepareCreate("test")
+                .setSettings(Settings.builder().put(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING.getKey(), 5))
+                .setMapping("""
+                    {
+                      "dynamic": "runtime",
+                      "properties": {
+                        "runtime": {
+                          "type": "object"
+                        },
+                        "mapped_obj": {
+                          "type": "object",
+                          "dynamic": "true_until_limit"
+                        }
+                      }
+                    }""")
+                .get()
+        );
+
+        client().index(
+            new IndexRequest("test").setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
+                .source(orderedMap("dynamic.keyword", "foo", "mapped_obj.number", 1, "mapped_obj.string", "foo"))
+        ).get();
+
+        var fields = client().prepareSearch("test").setQuery(new MatchAllQueryBuilder()).addFetchField("*").get().getHits().getHits()[0]
+            .getFields();
+        assertThat(fields.keySet(), equalTo(Set.of("dynamic.keyword", "mapped_obj.number", "_ignored")));
+        assertThat(fields.get("dynamic.keyword").getValues(), equalTo(List.of("foo")));
+        assertThat(fields.get("mapped_obj.number").getValues(), equalTo(List.of(1L)));
+        assertThat(fields.get("_ignored").getValues(), equalTo(List.of("mapped_obj.string")));
+    }
+
+    private LinkedHashMap<String, Object> orderedMap(Object... entries) {
+        var map = new LinkedHashMap<String, Object>();
+        for (int i = 0; i < entries.length; i += 2) {
+            map.put((String) entries[i], entries[i + 1]);
+        }
+        return map;
+    }
+
+    private SearchHit indexIgnoreDynamicBeyond(int fieldLimit, Map<String, Object> source) throws Exception {
+        prepareCreate("index", Settings.builder().put(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING.getKey(), fieldLimit)).setMapping(
+            Map.of("dynamic", "true_until_limit")
+        ).get();
+        ensureGreen("index");
+        client().prepareIndex("index").setId("1").setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE).setSource(source).get();
+        return client().prepareSearch("index").setQuery(new MatchAllQueryBuilder()).addFetchField("*").get().getHits().getHits()[0];
+    }
+
     public void testTotalFieldsLimitWithRuntimeFields() {
         Settings indexSettings = Settings.builder()
             .put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1)