packages/prometheus: Add metrics_count to collector and remote_writ…

…e if `metrics_count: true` (elastic#11447) * Add to collector and remote_write * Update packages/prometheus/changelog.yml * Update packages/prometheus/changelog.yml * Use better field descriptions * Update packages/prometheus/data_stream/remote_write/manifest.yml Co-authored-by: Gabriel Pop <[email protected]> * Update packages/prometheus/data_stream/collector/manifest.yml * Update packages/prometheus/data_stream/collector/fields/fields.yml * Address review comments * Update packages/prometheus/data_stream/collector/fields/fields.yml Co-authored-by: Ishleen Kaur <[email protected]> * Apply suggestions from code review Co-authored-by: Ishleen Kaur <[email protected]> * Update packages/prometheus/data_stream/collector/manifest.yml * Update packages/prometheus/data_stream/remote_write/manifest.yml * Address review comments * Address review comments --------- Co-authored-by: Gabriel Pop <[email protected]> Co-authored-by: Ishleen Kaur <[email protected]>
chemamartinez · Oct 23, 2024 · adbd663 · adbd663
1 parent 1a6fe15
commit adbd663
Show file tree

Hide file tree

Showing 10 changed files with 53 additions and 3 deletions.
diff --git a/packages/prometheus/_dev/build/docs/README.md b/packages/prometheus/_dev/build/docs/README.md
@@ -101,6 +101,10 @@ When `Use Types` and `Rate Counters` are enabled, metrics are stored like this:
 }
 ```
 
+#### Metrics count
+
+The Prometheus integration's `collector` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.
+
 #### Scraping all metrics from a Prometheus server
 
 We recommend using the Remote Write dataset for this, and make Prometheus push metrics to Agent.
@@ -321,6 +325,10 @@ Note that when using `types_patterns`, the provided patterns have higher priorit
 For instance if `_histogram_total` is a defined histogram pattern, then a metric like `network_bytes_histogram_total`
 will be handled as a histogram, even if it has the suffix `_total` which is a default pattern for counters.
 
+#### Metrics count
+
+The Prometheus integration's `remote_write` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.
+
 ### Prometheus Queries (PromQL)
 
 The Prometheus `query` dataset executes specific Prometheus queries against [Promethes Query API](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries).

diff --git a/packages/prometheus/changelog.yml b/packages/prometheus/changelog.yml
@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.20.0"
+  changes:
+    - description: Add support for `metrics_count` to count number of metrics per Elasticsearch document.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11447
 - version: "1.19.0"
   changes:
     - description: Allow automatic rerouting in promethues ingest-pipelines. E.g. for rerouting of events in Kubernetes environments.

diff --git a/packages/prometheus/data_stream/collector/agent/stream/stream.yml.hbs b/packages/prometheus/data_stream/collector/agent/stream/stream.yml.hbs
@@ -14,6 +14,7 @@ metrics_filters.include:
 metrics_path: {{metrics_path}}
 period: {{period}}
 rate_counters: {{rate_counters}}
+metrics_count: {{metrics_count}}
 {{#if bearer_token_file}}
 bearer_token_file: {{bearer_token_file}}
 ssl.verification_mode: {{ssl.verification_mode}}

diff --git a/packages/prometheus/data_stream/collector/fields/fields.yml b/packages/prometheus/data_stream/collector/fields/fields.yml
@@ -1,3 +1,7 @@
+- name: metrics_count
+  type: long
+  description: |
+    Total count of Prometheus metrics within the Elasticsearch document. This value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.
 - name: prometheus
   type: group
   fields:

diff --git a/packages/prometheus/data_stream/collector/manifest.yml b/packages/prometheus/data_stream/collector/manifest.yml
@@ -169,6 +169,15 @@ streams:
         default: prometheus.collector
         required: true
         show_user: true
+      - name: metrics_count
+        type: bool
+        title: Metrics Count
+        multi: false
+        required: false
+        show_user: false
+        default: false
+        description: >
+          Enable `Metrics Count` to get the total number of Prometheus metrics contained within the Elasticsearch document.
       - name: processors
         type: yaml
         title: Processors
@@ -181,4 +190,3 @@ streams:
     title: Prometheus collector metrics
     enabled: true
     description: Collect Prometheus collector metrics
-
diff --git a/packages/prometheus/data_stream/remote_write/agent/stream/stream.yml.hbs b/packages/prometheus/data_stream/remote_write/agent/stream/stream.yml.hbs
@@ -11,6 +11,7 @@ ssl.certificate_authorities:
 {{/each}}
 {{/if}}
 period: {{period}}
+metrics_count: {{metrics_count}}
 rate_counters: {{rate_counters}}
 use_types: {{use_types}}
 types_patterns.exclude:

diff --git a/packages/prometheus/data_stream/remote_write/fields/fields.yml b/packages/prometheus/data_stream/remote_write/fields/fields.yml
@@ -1,3 +1,7 @@
+- name: metrics_count
+  type: long
+  description: |
+    Total count of Prometheus metrics within the Elasticsearch document. This value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.
 - name: prometheus
   type: group
   fields:

diff --git a/packages/prometheus/data_stream/remote_write/manifest.yml b/packages/prometheus/data_stream/remote_write/manifest.yml
@@ -93,6 +93,15 @@ streams:
         default: prometheus.remote_write
         required: true
         show_user: true
+      - name: metrics_count
+        type: bool
+        title: Metrics Count
+        multi: false
+        required: false
+        show_user: false
+        default: false
+        description: >
+          Enable `Metrics Count` to get the total number of Prometheus metrics contained within the Elasticsearch document.
       - name: processors
         type: yaml
         title: Processors

diff --git a/packages/prometheus/docs/README.md b/packages/prometheus/docs/README.md
@@ -101,6 +101,10 @@ When `Use Types` and `Rate Counters` are enabled, metrics are stored like this:
 }
 ```
 
+#### Metrics count
+
+The Prometheus integration's `collector` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.
+
 #### Scraping all metrics from a Prometheus server
 
 We recommend using the Remote Write dataset for this, and make Prometheus push metrics to Agent.
@@ -241,6 +245,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 | host.name | Name of the host.  It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |  |
 | host.os.build | OS build information. | keyword |  |
 | host.os.codename | OS codename, if any. | keyword |  |
+| metrics_count | Total count of Prometheus metrics within the Elasticsearch document. This value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines. | long |  |
 | prometheus.\*.counter | Prometheus counter metric | object | counter |
 | prometheus.\*.histogram | Prometheus histogram metric | object |  |
 | prometheus.\*.rate | Prometheus rated counter metric | object | gauge |
@@ -437,6 +442,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 | host.name | Name of the host.  It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |  |
 | host.os.build | OS build information. | keyword |  |
 | host.os.codename | OS codename, if any. | keyword |  |
+| metrics_count | Total count of Prometheus metrics within the Elasticsearch document. This value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines. | long |  |
 | prometheus.\*.counter | Prometheus counter metric | object | counter |
 | prometheus.\*.histogram | Prometheus histogram metric | object |  |
 | prometheus.\*.rate | Prometheus rated counter metric | object | gauge |
@@ -528,6 +534,10 @@ Note that when using `types_patterns`, the provided patterns have higher priorit
 For instance if `_histogram_total` is a defined histogram pattern, then a metric like `network_bytes_histogram_total`
 will be handled as a histogram, even if it has the suffix `_total` which is a default pattern for counters.
 
+#### Metrics count
+
+The Prometheus integration's `remote_write` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.
+
 ### Prometheus Queries (PromQL)
 
 The Prometheus `query` dataset executes specific Prometheus queries against [Promethes Query API](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries).

diff --git a/packages/prometheus/manifest.yml b/packages/prometheus/manifest.yml
@@ -1,15 +1,15 @@
 format_version: 2.10.0
 name: prometheus
 title: Prometheus
-version: 1.19.0
+version: 1.20.0
 description: Collect metrics from Prometheus servers with Elastic Agent.
 type: integration
 categories:
   - observability
   - monitoring
   - containers
 conditions:
-  kibana.version: "^8.14.0"
+  kibana.version: "^8.16.0"
 screenshots:
   - src: /img/prometheus-server-overview.png
     title: Metricbeat Prometheus Overview