From 52de21349008b53841a357c664117c79d16de380 Mon Sep 17 00:00:00 2001 From: Anderson Queiroz Date: Mon, 18 Nov 2024 09:15:31 +0100 Subject: [PATCH 1/3] libbeat: increase total_fields.limit to 12500 (#41640) * libbeat: increase index template total_fields.limit to 12500 It increased the `index.mapping.total_fields.limit` from `10000` to `12500` in order to avoid ingestion failures caused by too many field in the index. Since 8.15.0 the limit started to be hit. The field count being exceeded is on the index, counting all mapped fields and the dynamic fields. That's why a small event might trigger the error, the event contains new fields to be mapped which would exceed the total field limit if mapped. (cherry picked from commit 42dd93b99b1088d5a5e5aec7a387333e8aa3d906) # Conflicts: # CHANGELOG.asciidoc --- CHANGELOG.asciidoc | 5 +++++ libbeat/template/load_test.go | 2 +- libbeat/template/template.go | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 3f80a200b180..dc4ee4ba7b26 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -18,6 +18,7 @@ https://github.com/elastic/beats/compare/v8.15.3\...v8.15.4[View commits] *Affecting all Beats* - Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use. {pull}41356[41356] +- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640] *Filebeat* @@ -121,6 +122,10 @@ https://github.com/elastic/beats/compare/v8.15.0\...v8.15.1[View commits] *Affecting all Beats* +<<<<<<< HEAD +======= +- Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file. +>>>>>>> 42dd93b99b (libbeat: increase total_fields.limit to 12500 (#41640)) - Beats stop publishing data after a network error unless restarted. Avoid upgrading to 8.15.1. Affected Beats log `Get \"https://${ELASTICSEARCH_HOST}:443\": context canceled` repeatedly. {issue}40705{40705} ==== Bugfixes diff --git a/libbeat/template/load_test.go b/libbeat/template/load_test.go index 8f6b1837d9a2..db82384539c9 100644 --- a/libbeat/template/load_test.go +++ b/libbeat/template/load_test.go @@ -170,7 +170,7 @@ func TestFileLoader_Load(t *testing.T) { "refresh_interval": "5s", "mapping": mapstr.M{ "total_fields": mapstr.M{ - "limit": 10000, + "limit": defaultTotalFieldsLimit, }, }, "query": mapstr.M{ diff --git a/libbeat/template/template.go b/libbeat/template/template.go index 5663a55c9cb5..f68b56987fdb 100644 --- a/libbeat/template/template.go +++ b/libbeat/template/template.go @@ -35,7 +35,7 @@ import ( var ( // Defaults used in the template defaultDateDetection = false - defaultTotalFieldsLimit = 10000 + defaultTotalFieldsLimit = 12500 defaultMaxDocvalueFieldsSearch = 200 defaultFields []string From dc06c690dc9ee8f4c7e6cf84c2b454b7d911ec5e Mon Sep 17 00:00:00 2001 From: Anderson Queiroz Date: Mon, 18 Nov 2024 11:41:14 +0100 Subject: [PATCH 2/3] adjust changelog --- CHANGELOG.asciidoc | 4 ---- CHANGELOG.next.asciidoc | 3 ++- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index dc4ee4ba7b26..df820ae1c53e 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -18,7 +18,6 @@ https://github.com/elastic/beats/compare/v8.15.3\...v8.15.4[View commits] *Affecting all Beats* - Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use. {pull}41356[41356] -- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640] *Filebeat* @@ -123,9 +122,6 @@ https://github.com/elastic/beats/compare/v8.15.0\...v8.15.1[View commits] *Affecting all Beats* <<<<<<< HEAD -======= -- Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file. ->>>>>>> 42dd93b99b (libbeat: increase total_fields.limit to 12500 (#41640)) - Beats stop publishing data after a network error unless restarted. Avoid upgrading to 8.15.1. Affected Beats log `Get \"https://${ELASTICSEARCH_HOST}:443\": context canceled` repeatedly. {issue}40705{40705} ==== Bugfixes diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 2ffaf3075fff..51249f2f57ed 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -15,6 +15,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff] - Filebeat now needs `dup3`, `faccessat2`, `prctl` and `setrlimit` syscalls to run the journald input. If this input is not being used, the syscalls are not needed. All Beats have those syscalls allowed now because the default seccomp policy is global to all Beats. {pull}40061[40061] - Beats will rate limit the logs about errors when indexing events on Elasticsearch, logging a summary every 10s. The logs sent to the event log is unchanged. {issue}40157[40157] - Drop support for Debian 10 and upgrade statically linked glibc from 2.28 to 2.31 {pull}41402[41402] +- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640] *Auditbeat* @@ -368,7 +369,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff] - Add metrics for the vSphere Virtualmachine metricset. {pull}40485[40485] - Log the total time taken for GCP `ListTimeSeries` and `AggregatedList` requests {pull}40661[40661] - Add metrics related to triggered alarms in all the vSphere metricsets. {pull}40714[40714] {pull}40876[40876] -- Add new metricset datastorecluster for vSphere module. {pull}40634[40634] +- Add new metricset datastorecluster for vSphere module. {pull}40634[40634] - Add support for new metrics in datastorecluster metricset. {pull}40694[40694] - Add metrics related to alert in all the vSphere metricsets. {pull}40714[40714] - Add new metrics fot datastore and minor changes to overall vSphere metrics {pull}40766[40766] From 5a2073a26186ceea139e33bbc4bfab69ba8fed37 Mon Sep 17 00:00:00 2001 From: Anderson Queiroz Date: Mon, 18 Nov 2024 11:42:47 +0100 Subject: [PATCH 3/3] fix changelog --- CHANGELOG.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index df820ae1c53e..3f80a200b180 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -121,7 +121,6 @@ https://github.com/elastic/beats/compare/v8.15.0\...v8.15.1[View commits] *Affecting all Beats* -<<<<<<< HEAD - Beats stop publishing data after a network error unless restarted. Avoid upgrading to 8.15.1. Affected Beats log `Get \"https://${ELASTICSEARCH_HOST}:443\": context canceled` repeatedly. {issue}40705{40705} ==== Bugfixes