Skip to content

[ECOINT-248] Redpanda: v2.2.0 (metrics update) #2747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[ECOINT-248] Redpanda: v2.2.0 (metrics update) #2747

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d46215f
Redpanda: v2.2.0 (metrics update)
pmw-rp Jul 20, 2025
c54a631
Adding trailing commas to list definitions and object definitions.
pmw-rp Aug 19, 2025
8e17795
Adding orientations where missing, redefining orientations throughout…
pmw-rp Aug 19, 2025
1c1b510
Merge branch 'master' into redpanda-v2.2.0
pmw-rp Aug 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions redpanda/CHANGELOG.md
View file
Open in desktop
Copy link
Collaborator

@dd-dominic dd-dominic Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the redpanda_cloud_client_dowload_backoff typo in the current metadata file (https://github.com/DataDog/integrations-extras/blob/master/redpanda/metadata.csv)

Copy link
Contributor Author

@pmw-rp pmw-rp Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typo is in metrics.py, not metadata.csv - so this would have affected the actual metrics produced, not the published "what metrics should you expect".

Original file line number Diff line number Diff line change
@@ -1,5 +1,44 @@
# CHANGELOG - redpanda

## 2.2.0 / 2025-07-15

Metrics update release.

***Added***:

* Added new default metrics groups:
* Security and TLS
* `redpanda.authorization`: +1 metric
* `redpanda.security`: +2 metrics
* `redpanda.tls`: +5 metrics

* Added new additional metrics groups:
* Apache Iceberg integration
* `redpanda.iceberg`: +32 metrics
* Redpanda Transforms and WASM
* `redpanda.transform`: +9 metrics
* `redpanda.wasm`: +4 metrics
* Debug bundles
* `redpanda.debug_bundle`: +4 metrics

* Added new metrics under existing default metrics groups:
* `redpanda.application`: +1 metric
* `redpanda.cluster`: +5 metrics
* `redpanda.rpc`: +2 metrics
* `redpanda.kafka`: +8 metrics
* `redpanda.raft`: +5 metrics
* `redpanda.storage`: +2 metrics

* Added new metrics under existing additional metrics groups:
* `redpanda.cloud`: +27 metrics
* `redpanda.pandaproxy`: +3 metrics
* `redpanda.schemaregistry`: +7 metrics

***Fixed***:

* Typo: `redpanda.schema_registry_latency_seconds` is now `redpanda.schema_registry.latency_seconds`
* Typo: `redpanda_cloud_client_dowload_backoff` is now `cloud.client_download_backoff`

Comment on lines +3 to +41
Copy link
Collaborator

@dd-dominic dd-dominic Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## 2.2.0 / 2025-07-15
Metrics update release.
***Added***:
* Added new default metrics groups:
* Security and TLS
* `redpanda.authorization`: +1 metric
* `redpanda.security`: +2 metrics
* `redpanda.tls`: +5 metrics
* Added new additional metrics groups:
* Apache Iceberg integration
* `redpanda.iceberg`: +32 metrics
* Redpanda Transforms and WASM
* `redpanda.transform`: +9 metrics
* `redpanda.wasm`: +4 metrics
* Debug bundles
* `redpanda.debug_bundle`: +4 metrics
* Added new metrics under existing default metrics groups:
* `redpanda.application`: +1 metric
* `redpanda.cluster`: +5 metrics
* `redpanda.rpc`: +2 metrics
* `redpanda.kafka`: +8 metrics
* `redpanda.raft`: +5 metrics
* `redpanda.storage`: +2 metrics
* Added new metrics under existing additional metrics groups:
* `redpanda.cloud`: +27 metrics
* `redpanda.pandaproxy`: +3 metrics
* `redpanda.schemaregistry`: +7 metrics
***Fixed***:
* Typo: `redpanda.schema_registry_latency_seconds` is now `redpanda.schema_registry.latency_seconds`
* Typo: `redpanda_cloud_client_dowload_backoff` is now `cloud.client_download_backoff`
## 2.2.0 / 2025-07-28
***Added***:
* New default metrics groups:
* Security and TLS
* `redpanda.authorization`: +1 metric
* `redpanda.security`: +2 metrics
* `redpanda.tls`: +5 metrics
* New additional metrics groups:
* Apache Iceberg integration
* `redpanda.iceberg`: +32 metrics
* Redpanda Transforms and WASM
* `redpanda.transform`: +9 metrics
* `redpanda.wasm`: +4 metrics
* Debug bundles
* `redpanda.debug_bundle`: +4 metrics
* New metrics under existing default metrics groups:
* `redpanda.application`: +1 metric
* `redpanda.cluster`: +5 metrics
* `redpanda.rpc`: +2 metrics
* `redpanda.kafka`: +8 metrics
* `redpanda.raft`: +5 metrics
* `redpanda.storage`: +2 metrics
* New metrics under existing additional metrics groups:
* `redpanda.cloud`: +27 metrics
* `redpanda.pandaproxy`: +3 metrics
* `redpanda.schemaregistry`: +7 metrics
***Fixed***:
* Typo: `redpanda.schema_registry_latency_seconds` is now `redpanda.schema_registry.latency_seconds`
* Typo: `redpanda_cloud_client_dowload_backoff` is now `cloud.client_download_backoff`

## 2.1.0 / 2025-06-18

***Added***:
Expand Down
2 changes: 1 addition & 1 deletion redpanda/README.md
View file
Open in desktop
Copy link
Collaborator

@dd-dominic dd-dominic Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace datadog-agent integration install -t datadog-redpanda==<INTEGRATION_VERSION> with datadog-agent integration install -t datadog-redpanda==2.2.0

Copy link
Contributor Author

@pmw-rp pmw-rp Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Collaborator

@dd-dominic dd-dominic Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmw-rp I'm not seeing the changes. Since this PR has been closed, you'll need to reopen this PR or open a new PR.

Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ To build an updated version of the Agent:
```dockerfile
FROM gcr.io/datadoghq/agent:latest

ARG INTEGRATION_VERSION=2.1.0
ARG INTEGRATION_VERSION=2.2.0

RUN agent integration install -r -t datadog-redpanda==${INTEGRATION_VERSION}
```
Expand Down
2 changes: 1 addition & 1 deletion redpanda/datadog_checks/redpanda/__about__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '2.1.0'
__version__ = '2.2.0'
146 changes: 144 additions & 2 deletions redpanda/datadog_checks/redpanda/metrics.py
View file
Open in desktop

Large diffs are not rendered by default.

270 changes: 191 additions & 79 deletions redpanda/metadata.csv
View file
Open in desktop
Copy link
Collaborator

@dd-dominic dd-dominic Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't remove metrics, so we'll need to keep the row for redpanda.schema_registry_latency_seconds. In the description for this row, please add "[supported in v2.1.0 and below]". Additionally, for the new row redpanda.schema_registry.latency_seconds, add "[supported in v2.2.0+]" in the description.

Can you confirm no additional metrics were removed with this update? It's difficult to see in the GitHub diff.

Large diffs are not rendered by default.

159 changes: 152 additions & 7 deletions redpanda/tests/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,22 @@
'redpanda.application': [
'redpanda.application.uptime',
'redpanda.application.build',
'redpanda.application.fips_mode'
],
'redpanda.controller': [
'redpanda.controller.log_limit_requests_available',
'redpanda.controller.log_limit_requests_dropped.count',
],
'redpanda.cloud': [
'redpanda.cloud.client_backoff.count',
'redpanda.cloud.client_pool_utilization',
'redpanda.cloud.client_download_backoff.count',
'redpanda.cloud.client_downloads.count',
'redpanda.cloud.client_lease_duration.count',
'redpanda.cloud.client_lease_duration.bucket',
'redpanda.cloud.client_lease_duration.sum',
'redpanda.cloud.client_not_found.count',
'redpanda.cloud.client_num_borrows.count',
'redpanda.cloud.client_upload_backoff.count',
'redpanda.cloud.client_uploads.count',
'redpanda.cloud.storage.active_segments',
Expand All @@ -37,7 +43,17 @@
'redpanda.cloud.storage.cache_op_miss.count',
'redpanda.cloud.storage.op_put.count',
'redpanda.cloud.storage.cache_space_files',
'redpanda.cloud.storage.cache_space_hwm_files',
'redpanda.cloud.storage.cache_space_hwm_size_bytes',
'redpanda.cloud.storage.cache_space_size_bytes',
'redpanda.cloud.storage.cache_space_tracker_size',
'redpanda.cloud.storage.cache_space_tracker_syncs.count',
'redpanda.cloud.storage_cache_trim_carryover_trims.count',
'redpanda.cloud.storage_cache_trim_exhaustive_trims.count',
'redpanda.cloud.storage_cache_trim_failed_trims.count',
'redpanda.cloud.storage_cache_trim_fast_trims.count',
'redpanda.cloud.storage_cache_trim_in_mem_trims.count',
'redpanda.cloud.storage_cloud_log_size',
'redpanda.cloud.storage.deleted_segments.count',
'redpanda.cloud.storage.errors.count',
'redpanda.cloud.storage.housekeeping.drains',
Expand All @@ -46,44 +62,84 @@
'redpanda.cloud.storage.housekeeping.jobs_skipped.count',
'redpanda.cloud.storage.housekeeping.pauses',
'redpanda.cloud.storage.housekeeping.resumes',
'redpanda.cloud.storage_housekeeping_requests_throttled_average_rate',
'redpanda.cloud.storage.housekeeping.rounds.count',
'redpanda.cloud.storage.jobs.cloud_segment_reuploads',
'redpanda.cloud.storage.jobs.local_segment_reuploads',
'redpanda.cloud.storage.jobs.manifest_reuploads',
'redpanda.cloud.storage.jobs.metadata_syncs',
'redpanda.cloud.storage.jobs.segment_deletions',
'redpanda.cloud.storage_limits_downloads_throttled_sum.count',
'redpanda.cloud.storage_partition_manifest_uploads.count',
'redpanda.cloud.storage_partition_readers',
'redpanda.cloud.storage_partition_readers_delayed.count',
'redpanda.cloud.storage_paused_archivers',
'redpanda.cloud.storage.readers',
'redpanda.cloud.storage_segment_index_uploads.count',
'redpanda.cloud.storage_segment_materializations_delayed.count',
'redpanda.cloud.storage_segment_readers_delayed.count',
'redpanda.cloud.storage_segment_uploads.count',
'redpanda.cloud.storage.segments',
'redpanda.cloud.storage.segments_pending_deletion',
'redpanda.cloud.storage_spillover_manifest_uploads.count',
'redpanda.cloud.storage_spillover_manifests_materialized_bytes',
'redpanda.cloud.storage_spillover_manifests_materialized_count',
'redpanda.cloud.storage.uploaded_bytes.count',
],
'redpanda.cluster': [
'redpanda.cluster.brokers',
'redpanda.cluster.features_enterprise_license_expiry_sec',
'redpanda.cluster.latest_cluster_metadata_manifest_age',
'redpanda.cluster.members_backend_queued_node_operations',
'redpanda.cluster.non_homogenous_fips_mode',
'redpanda.cluster.partition_num_with_broken_rack_constraint',
'redpanda.cluster.partition_schema_id_validation_records_failed.count',
'redpanda.cluster.partitions',
'redpanda.cluster.topics',
'redpanda.cluster.unavailable_partitions',
],
'redpanda.debug_bundle': [
'redpanda.debug_bundle.failed_generation_count.count',
'redpanda.debug_bundle.last_failed_bundle_timestamp_seconds',
'redpanda.debug_bundle.last_successful_bundle_timestamp_seconds',
'redpanda.debug_bundle.successful_generation_count.count',
],
'redpanda.rpc': [
'redpanda.rpc.active_connections',
'redpanda.rpc.received_bytes.count',
'redpanda.rpc.request_errors.count',
'redpanda.rpc.request_latency_seconds.count',
'redpanda.rpc.request_latency_seconds.bucket',
'redpanda.rpc.request_latency_seconds.sum',
'redpanda.rpc.sent_bytes.count',
],
'redpanda.io_queue': [
'redpanda.io_queue.total_read_ops.count',
'redpanda.io_queue.total_write_ops.count',
],
'redpanda.kafka': [
'redpanda.kafka.handler_latency_seconds.sum',
'redpanda.kafka.handler_latency_seconds.bucket',
'redpanda.kafka.handler_latency_seconds.count',
'redpanda.kafka.partition_committed_offset',
'redpanda.kafka.partitions',
'redpanda.kafka.quotas_client_quota_throttle_time.sum',
'redpanda.kafka.quotas_client_quota_throttle_time.bucket',
'redpanda.kafka.quotas_client_quota_throttle_time.count',
'redpanda.kafka.quotas_client_quota_throughput.sum',
'redpanda.kafka.quotas_client_quota_throughput.bucket',
'redpanda.kafka.quotas_client_quota_throughput.count',
'redpanda.kafka.records_fetched.count',
'redpanda.kafka.records_produced.count',
'redpanda.kafka.replicas',
'redpanda.kafka.request_latency_seconds.sum',
'redpanda.kafka.request_latency_seconds.bucket',
'redpanda.kafka.request_latency_seconds.count',
'redpanda.kafka.request_bytes.count',
'redpanda.kafka.rpc_sasl_session_expiration.count',
'redpanda.kafka.rpc_sasl_session_reauth_attempts.count',
'redpanda.kafka.rpc_sasl_session_revoked.count',
'redpanda.kafka.under_replicated_replicas',
'redpanda.kafka.partition_committed_offset',
'redpanda.kafka.partitions',
'redpanda.kafka.replicas',
],
'redpanda.kafka.consumer_group_info': [
'redpanda.kafka.group_count',
Expand All @@ -108,8 +164,11 @@
'redpanda.node_status.rpcs_timed_out',
],
'redpanda.pandaproxy': [
'redpanda.pandaproxy.request_latency.count',
'redpanda.pandaproxy.inflight_requests_memory_usage_ratio',
'redpanda.pandaproxy.inflight_requests_usage_ratio',
'redpanda.pandaproxy.queued_requests_memory_blocked',
'redpanda.pandaproxy.request_errors.count',
'redpanda.pandaproxy.request_latency.count',
'redpanda.pandaproxy.request_latency.bucket',
'redpanda.pandaproxy.request_latency.sum',
],
Expand All @@ -120,7 +179,12 @@
],
'redpanda.raft': [
'redpanda.raft.leadership_changes.count',
'redpanda.raft.learners_gap_bytes',
'redpanda.raft.recovery_offsets_pending',
'redpanda.raft.recovery_bandwidth',
'redpanda.raft.recovery_consumed_bandwidth',
'redpanda.raft.recovery_partitions_active',
'redpanda.raft.recovery_partitions_to_recover',
],
'redpanda.reactor': [
'redpanda.reactor.cpu_busy_seconds',
Expand All @@ -129,21 +193,96 @@
'redpanda.scheduler.runtime_seconds.count',
],
'redpanda.schemaregistry': [
'redpanda.schema_registry.cache_schema_count',
'redpanda.schema_registry.cache_schema_memory_bytes',
'redpanda.schema_registry.cache_subject_count',
'redpanda.schema_registry.cache_subject_version_count',
'redpanda.schema_registry.inflight_requests_memory_usage_ratio',
'redpanda.schema_registry.inflight_requests_usage_ratio',
'redpanda.schema_registry.queued_requests_memory_blocked',
'redpanda.schema_registry.errors.count',
'redpanda.schema_registry_latency_seconds.count',
'redpanda.schema_registry_latency_seconds.bucket',
'redpanda.schema_registry_latency_seconds.sum',
'redpanda.schema_registry.latency_seconds.count',
'redpanda.schema_registry.latency_seconds.sum',
'redpanda.schema_registry.latency_seconds.bucket',
],
'redpanda.storage': [
'redpanda.storage.cache_disk_free_bytes',
'redpanda.storage.cache_disk_free_space_alert',
'redpanda.storage.disk_free_bytes',
'redpanda.storage.disk_free_space_alert',
'redpanda.storage.disk_total_bytes',
],
'redpanda.iceberg': [
'redpanda.iceberg.rest_client_active_gets',
'redpanda.iceberg.rest_client_active_puts',
'redpanda.iceberg.rest_client_active_requests',
'redpanda.iceberg.rest_client_num_commit_table_update_requests.count',
'redpanda.iceberg.rest_client_num_commit_table_update_requests_failed.count',
'redpanda.iceberg.rest_client_num_create_namespace_requests.count',
'redpanda.iceberg.rest_client_num_create_namespace_requests_failed.count',
'redpanda.iceberg.rest_client_num_create_table_requests.count',
'redpanda.iceberg.rest_client_num_create_table_requests_failed.count',
'redpanda.iceberg.rest_client_num_drop_table_requests.count',
'redpanda.iceberg.rest_client_num_drop_table_requests_failed.count',
'redpanda.iceberg.rest_client_num_get_config_requests.count',
'redpanda.iceberg.rest_client_num_get_config_requests_failed.count',
'redpanda.iceberg.rest_client_num_load_table_requests.count',
'redpanda.iceberg.rest_client_num_load_table_requests_failed.count',
'redpanda.iceberg.rest_client_num_oauth_token_requests.count',
'redpanda.iceberg.rest_client_num_oauth_token_requests_failed.count',
'redpanda.iceberg.rest_client_num_request_timeouts.count',
'redpanda.iceberg.rest_client_num_transport_errors.count',
'redpanda.iceberg.rest_client_total_gets.count',
'redpanda.iceberg.rest_client_total_inbound_bytes.count',
'redpanda.iceberg.rest_client_total_outbound_bytes.count',
'redpanda.iceberg.rest_client_total_puts.count',
'redpanda.iceberg.rest_client_total_requests.count',
'redpanda.iceberg.translation_decompressed_bytes_processed.count',
'redpanda.iceberg.translation_dlq_files_created.count',
'redpanda.iceberg.translation_files_created.count',
'redpanda.iceberg.translation_invalid_records.count',
'redpanda.iceberg.translation_parquet_bytes_added.count',
'redpanda.iceberg.translation_parquet_rows_added.count',
'redpanda.iceberg.translation_raw_bytes_processed.count',
'redpanda.iceberg.translation_translations_finished.count',
],
'redpanda.transform': [
'redpanda.transform.execution_errors.count',
'redpanda.transform.execution_latency_sec.count',
'redpanda.transform.execution_latency_sec.bucket',
'redpanda.transform.execution_latency_sec.sum',
'redpanda.transform.failures.count',
'redpanda.transform.processor_lag.count',
'redpanda.transform.read_bytes.count',
'redpanda.transform.state',
'redpanda.transform.write_bytes.count',
],
'redpanda.wasm': [
'redpanda.wasm.binary_executable_memory_usage',
'redpanda.wasm.engine_cpu_seconds.count',
'redpanda.wasm.engine_max_memory',
'redpanda.wasm.engine_memory_usage',
],
'redpanda.tls': [
'redpanda.tls.certificate_expires_at_timestamp_seconds',
'redpanda.tls.certificate_serial',
'redpanda.tls.certificate_valid',
'redpanda.tls.loaded_at_timestamp_seconds',
'redpanda.tls.truststore_expires_at_timestamp_seconds',
],
'redpanda.authorization': [
'redpanda.authorization.result.count',
],
'redpanda.security': [
'redpanda.security.audit_errors.count',
'redpanda.security.audit_last_event_timestamp_seconds.count',
]
}
# fmt: on

INSTANCE_DEFAULT_GROUPS = [
'redpanda.application',
'redpanda.authorization',
'redpanda.cluster',
'redpanda.io_queue',
'redpanda.kafka',
Expand All @@ -155,16 +294,22 @@
'redpanda.raft',
'redpanda.reactor',
'redpanda.rpc',
'redpanda.security',
'redpanda.storage',
'redpanda.tls'
]

INSTANCE_ADDITIONAL_GROUPS = [
'redpanda.cloud',
'redpanda.controller',
'redpanda.debug_bundle',
'redpanda.node_status',
'redpanda.pandaproxy',
'redpanda.scheduler',
'redpanda.schemaregistry',
'redpanda.iceberg',
'redpanda.transform',
'redpanda.wasm'
]


Expand Down
Loading
Loading