Skip to content

[ECOINT-248] Redpanda: v2.2.0 (metrics update) #2747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[ECOINT-248] Redpanda: v2.2.0 (metrics update) #2747

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d46215f
Redpanda: v2.2.0 (metrics update)
pmw-rp Jul 20, 2025
c54a631
Adding trailing commas to list definitions and object definitions.
pmw-rp Aug 19, 2025
8e17795
Adding orientations where missing, redefining orientations throughout…
pmw-rp Aug 19, 2025
1c1b510
Merge branch 'master' into redpanda-v2.2.0
pmw-rp Aug 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions redpanda/CHANGELOG.md
View file
Open in desktop
Copy link
Collaborator

@dd-dominic dd-dominic Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the redpanda_cloud_client_dowload_backoff typo in the current metadata file (https://github.com/DataDog/integrations-extras/blob/master/redpanda/metadata.csv)

Copy link
Contributor Author

@pmw-rp pmw-rp Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typo is in metrics.py, not metadata.csv - so this would have affected the actual metrics produced, not the published "what metrics should you expect".

Original file line number Diff line number Diff line change
@@ -1,5 +1,44 @@
# CHANGELOG - redpanda

## 2.2.0 / 2025-07-15

Metrics update release.

***Added***:

* Added new default metrics groups:
* Security and TLS
* `redpanda.authorization`: +1 metric
* `redpanda.security`: +2 metrics
* `redpanda.tls`: +5 metrics

* Added new additional metrics groups:
* Apache Iceberg integration
* `redpanda.iceberg`: +32 metrics
* Redpanda Transforms and WASM
* `redpanda.transform`: +9 metrics
* `redpanda.wasm`: +4 metrics
* Debug bundles
* `redpanda.debug_bundle`: +4 metrics

* Added new metrics under existing default metrics groups:
* `redpanda.application`: +1 metric
* `redpanda.cluster`: +5 metrics
* `redpanda.rpc`: +2 metrics
* `redpanda.kafka`: +8 metrics
* `redpanda.raft`: +5 metrics
* `redpanda.storage`: +2 metrics

* Added new metrics under existing additional metrics groups:
* `redpanda.cloud`: +27 metrics
* `redpanda.pandaproxy`: +3 metrics
* `redpanda.schemaregistry`: +7 metrics

***Fixed***:

* Typo: `redpanda.schema_registry_latency_seconds` is now `redpanda.schema_registry.latency_seconds`
* Typo: `redpanda_cloud_client_dowload_backoff` is now `cloud.client_download_backoff`

Comment on lines +3 to +41
Copy link
Collaborator

@dd-dominic dd-dominic Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## 2.2.0 / 2025-07-15
Metrics update release.
***Added***:
* Added new default metrics groups:
* Security and TLS
* `redpanda.authorization`: +1 metric
* `redpanda.security`: +2 metrics
* `redpanda.tls`: +5 metrics
* Added new additional metrics groups:
* Apache Iceberg integration
* `redpanda.iceberg`: +32 metrics
* Redpanda Transforms and WASM
* `redpanda.transform`: +9 metrics
* `redpanda.wasm`: +4 metrics
* Debug bundles
* `redpanda.debug_bundle`: +4 metrics
* Added new metrics under existing default metrics groups:
* `redpanda.application`: +1 metric
* `redpanda.cluster`: +5 metrics
* `redpanda.rpc`: +2 metrics
* `redpanda.kafka`: +8 metrics
* `redpanda.raft`: +5 metrics
* `redpanda.storage`: +2 metrics
* Added new metrics under existing additional metrics groups:
* `redpanda.cloud`: +27 metrics
* `redpanda.pandaproxy`: +3 metrics
* `redpanda.schemaregistry`: +7 metrics
***Fixed***:
* Typo: `redpanda.schema_registry_latency_seconds` is now `redpanda.schema_registry.latency_seconds`
* Typo: `redpanda_cloud_client_dowload_backoff` is now `cloud.client_download_backoff`
## 2.2.0 / 2025-07-28
***Added***:
* New default metrics groups:
* Security and TLS
* `redpanda.authorization`: +1 metric
* `redpanda.security`: +2 metrics
* `redpanda.tls`: +5 metrics
* New additional metrics groups:
* Apache Iceberg integration
* `redpanda.iceberg`: +32 metrics
* Redpanda Transforms and WASM
* `redpanda.transform`: +9 metrics
* `redpanda.wasm`: +4 metrics
* Debug bundles
* `redpanda.debug_bundle`: +4 metrics
* New metrics under existing default metrics groups:
* `redpanda.application`: +1 metric
* `redpanda.cluster`: +5 metrics
* `redpanda.rpc`: +2 metrics
* `redpanda.kafka`: +8 metrics
* `redpanda.raft`: +5 metrics
* `redpanda.storage`: +2 metrics
* New metrics under existing additional metrics groups:
* `redpanda.cloud`: +27 metrics
* `redpanda.pandaproxy`: +3 metrics
* `redpanda.schemaregistry`: +7 metrics
***Fixed***:
* Typo: `redpanda.schema_registry_latency_seconds` is now `redpanda.schema_registry.latency_seconds`
* Typo: `redpanda_cloud_client_dowload_backoff` is now `cloud.client_download_backoff`

## 2.1.0 / 2025-06-18

***Added***:
Expand Down
2 changes: 1 addition & 1 deletion redpanda/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ To build an updated version of the Agent:
```dockerfile
FROM gcr.io/datadoghq/agent:latest

ARG INTEGRATION_VERSION=2.1.0
ARG INTEGRATION_VERSION=2.2.0

RUN agent integration install -r -t datadog-redpanda==${INTEGRATION_VERSION}
```
Expand Down
2 changes: 1 addition & 1 deletion redpanda/datadog_checks/redpanda/__about__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '2.1.0'
__version__ = '2.2.0'
146 changes: 144 additions & 2 deletions redpanda/datadog_checks/redpanda/metrics.py
View file
Open in desktop

Large diffs are not rendered by default.

270 changes: 191 additions & 79 deletions redpanda/metadata.csv
View file
Open in desktop

Large diffs are not rendered by default.

159 changes: 152 additions & 7 deletions redpanda/tests/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,22 @@
'redpanda.application': [
'redpanda.application.uptime',
'redpanda.application.build',
'redpanda.application.fips_mode',
],
'redpanda.controller': [
'redpanda.controller.log_limit_requests_available',
'redpanda.controller.log_limit_requests_dropped.count',
],
'redpanda.cloud': [
'redpanda.cloud.client_backoff.count',
'redpanda.cloud.client_pool_utilization',
'redpanda.cloud.client_download_backoff.count',
'redpanda.cloud.client_downloads.count',
'redpanda.cloud.client_lease_duration.count',
'redpanda.cloud.client_lease_duration.bucket',
'redpanda.cloud.client_lease_duration.sum',
'redpanda.cloud.client_not_found.count',
'redpanda.cloud.client_num_borrows.count',
'redpanda.cloud.client_upload_backoff.count',
'redpanda.cloud.client_uploads.count',
'redpanda.cloud.storage.active_segments',
Expand All @@ -37,7 +43,17 @@
'redpanda.cloud.storage.cache_op_miss.count',
'redpanda.cloud.storage.op_put.count',
'redpanda.cloud.storage.cache_space_files',
'redpanda.cloud.storage.cache_space_hwm_files',
'redpanda.cloud.storage.cache_space_hwm_size_bytes',
'redpanda.cloud.storage.cache_space_size_bytes',
'redpanda.cloud.storage.cache_space_tracker_size',
'redpanda.cloud.storage.cache_space_tracker_syncs.count',
'redpanda.cloud.storage_cache_trim_carryover_trims.count',
'redpanda.cloud.storage_cache_trim_exhaustive_trims.count',
'redpanda.cloud.storage_cache_trim_failed_trims.count',
'redpanda.cloud.storage_cache_trim_fast_trims.count',
'redpanda.cloud.storage_cache_trim_in_mem_trims.count',
'redpanda.cloud.storage_cloud_log_size',
'redpanda.cloud.storage.deleted_segments.count',
'redpanda.cloud.storage.errors.count',
'redpanda.cloud.storage.housekeeping.drains',
Expand All @@ -46,44 +62,84 @@
'redpanda.cloud.storage.housekeeping.jobs_skipped.count',
'redpanda.cloud.storage.housekeeping.pauses',
'redpanda.cloud.storage.housekeeping.resumes',
'redpanda.cloud.storage_housekeeping_requests_throttled_average_rate',
'redpanda.cloud.storage.housekeeping.rounds.count',
'redpanda.cloud.storage.jobs.cloud_segment_reuploads',
'redpanda.cloud.storage.jobs.local_segment_reuploads',
'redpanda.cloud.storage.jobs.manifest_reuploads',
'redpanda.cloud.storage.jobs.metadata_syncs',
'redpanda.cloud.storage.jobs.segment_deletions',
'redpanda.cloud.storage_limits_downloads_throttled_sum.count',
'redpanda.cloud.storage_partition_manifest_uploads.count',
'redpanda.cloud.storage_partition_readers',
'redpanda.cloud.storage_partition_readers_delayed.count',
'redpanda.cloud.storage_paused_archivers',
'redpanda.cloud.storage.readers',
'redpanda.cloud.storage_segment_index_uploads.count',
'redpanda.cloud.storage_segment_materializations_delayed.count',
'redpanda.cloud.storage_segment_readers_delayed.count',
'redpanda.cloud.storage_segment_uploads.count',
'redpanda.cloud.storage.segments',
'redpanda.cloud.storage.segments_pending_deletion',
'redpanda.cloud.storage_spillover_manifest_uploads.count',
'redpanda.cloud.storage_spillover_manifests_materialized_bytes',
'redpanda.cloud.storage_spillover_manifests_materialized_count',
'redpanda.cloud.storage.uploaded_bytes.count',
],
'redpanda.cluster': [
'redpanda.cluster.brokers',
'redpanda.cluster.features_enterprise_license_expiry_sec',
'redpanda.cluster.latest_cluster_metadata_manifest_age',
'redpanda.cluster.members_backend_queued_node_operations',
'redpanda.cluster.non_homogenous_fips_mode',
'redpanda.cluster.partition_num_with_broken_rack_constraint',
'redpanda.cluster.partition_schema_id_validation_records_failed.count',
'redpanda.cluster.partitions',
'redpanda.cluster.topics',
'redpanda.cluster.unavailable_partitions',
],
'redpanda.debug_bundle': [
'redpanda.debug_bundle.failed_generation_count.count',
'redpanda.debug_bundle.last_failed_bundle_timestamp_seconds',
'redpanda.debug_bundle.last_successful_bundle_timestamp_seconds',
'redpanda.debug_bundle.successful_generation_count.count',
],
'redpanda.rpc': [
'redpanda.rpc.active_connections',
'redpanda.rpc.received_bytes.count',
'redpanda.rpc.request_errors.count',
'redpanda.rpc.request_latency_seconds.count',
'redpanda.rpc.request_latency_seconds.bucket',
'redpanda.rpc.request_latency_seconds.sum',
'redpanda.rpc.sent_bytes.count',
],
'redpanda.io_queue': [
'redpanda.io_queue.total_read_ops.count',
'redpanda.io_queue.total_write_ops.count',
],
'redpanda.kafka': [
'redpanda.kafka.handler_latency_seconds.sum',
'redpanda.kafka.handler_latency_seconds.bucket',
'redpanda.kafka.handler_latency_seconds.count',
'redpanda.kafka.partition_committed_offset',
'redpanda.kafka.partitions',
'redpanda.kafka.quotas_client_quota_throttle_time.sum',
'redpanda.kafka.quotas_client_quota_throttle_time.bucket',
'redpanda.kafka.quotas_client_quota_throttle_time.count',
'redpanda.kafka.quotas_client_quota_throughput.sum',
'redpanda.kafka.quotas_client_quota_throughput.bucket',
'redpanda.kafka.quotas_client_quota_throughput.count',
'redpanda.kafka.records_fetched.count',
'redpanda.kafka.records_produced.count',
'redpanda.kafka.replicas',
'redpanda.kafka.request_latency_seconds.sum',
'redpanda.kafka.request_latency_seconds.bucket',
'redpanda.kafka.request_latency_seconds.count',
'redpanda.kafka.request_bytes.count',
'redpanda.kafka.rpc_sasl_session_expiration.count',
'redpanda.kafka.rpc_sasl_session_reauth_attempts.count',
'redpanda.kafka.rpc_sasl_session_revoked.count',
'redpanda.kafka.under_replicated_replicas',
'redpanda.kafka.partition_committed_offset',
'redpanda.kafka.partitions',
'redpanda.kafka.replicas',
],
'redpanda.kafka.consumer_group_info': [
'redpanda.kafka.group_count',
Expand All @@ -108,8 +164,11 @@
'redpanda.node_status.rpcs_timed_out',
],
'redpanda.pandaproxy': [
'redpanda.pandaproxy.request_latency.count',
'redpanda.pandaproxy.inflight_requests_memory_usage_ratio',
'redpanda.pandaproxy.inflight_requests_usage_ratio',
'redpanda.pandaproxy.queued_requests_memory_blocked',
'redpanda.pandaproxy.request_errors.count',
'redpanda.pandaproxy.request_latency.count',
'redpanda.pandaproxy.request_latency.bucket',
'redpanda.pandaproxy.request_latency.sum',
],
Expand All @@ -120,7 +179,12 @@
],
'redpanda.raft': [
'redpanda.raft.leadership_changes.count',
'redpanda.raft.learners_gap_bytes',
'redpanda.raft.recovery_offsets_pending',
'redpanda.raft.recovery_bandwidth',
'redpanda.raft.recovery_consumed_bandwidth',
'redpanda.raft.recovery_partitions_active',
'redpanda.raft.recovery_partitions_to_recover',
],
'redpanda.reactor': [
'redpanda.reactor.cpu_busy_seconds',
Expand All @@ -129,21 +193,96 @@
'redpanda.scheduler.runtime_seconds.count',
],
'redpanda.schemaregistry': [
'redpanda.schema_registry.cache_schema_count',
'redpanda.schema_registry.cache_schema_memory_bytes',
'redpanda.schema_registry.cache_subject_count',
'redpanda.schema_registry.cache_subject_version_count',
'redpanda.schema_registry.inflight_requests_memory_usage_ratio',
'redpanda.schema_registry.inflight_requests_usage_ratio',
'redpanda.schema_registry.queued_requests_memory_blocked',
'redpanda.schema_registry.errors.count',
'redpanda.schema_registry_latency_seconds.count',
'redpanda.schema_registry_latency_seconds.bucket',
'redpanda.schema_registry_latency_seconds.sum',
'redpanda.schema_registry.latency_seconds.count',
'redpanda.schema_registry.latency_seconds.sum',
'redpanda.schema_registry.latency_seconds.bucket',
],
'redpanda.storage': [
'redpanda.storage.cache_disk_free_bytes',
'redpanda.storage.cache_disk_free_space_alert',
'redpanda.storage.disk_free_bytes',
'redpanda.storage.disk_free_space_alert',
'redpanda.storage.disk_total_bytes',
],
'redpanda.iceberg': [
'redpanda.iceberg.rest_client_active_gets',
'redpanda.iceberg.rest_client_active_puts',
'redpanda.iceberg.rest_client_active_requests',
'redpanda.iceberg.rest_client_num_commit_table_update_requests.count',
'redpanda.iceberg.rest_client_num_commit_table_update_requests_failed.count',
'redpanda.iceberg.rest_client_num_create_namespace_requests.count',
'redpanda.iceberg.rest_client_num_create_namespace_requests_failed.count',
'redpanda.iceberg.rest_client_num_create_table_requests.count',
'redpanda.iceberg.rest_client_num_create_table_requests_failed.count',
'redpanda.iceberg.rest_client_num_drop_table_requests.count',
'redpanda.iceberg.rest_client_num_drop_table_requests_failed.count',
'redpanda.iceberg.rest_client_num_get_config_requests.count',
'redpanda.iceberg.rest_client_num_get_config_requests_failed.count',
'redpanda.iceberg.rest_client_num_load_table_requests.count',
'redpanda.iceberg.rest_client_num_load_table_requests_failed.count',
'redpanda.iceberg.rest_client_num_oauth_token_requests.count',
'redpanda.iceberg.rest_client_num_oauth_token_requests_failed.count',
'redpanda.iceberg.rest_client_num_request_timeouts.count',
'redpanda.iceberg.rest_client_num_transport_errors.count',
'redpanda.iceberg.rest_client_total_gets.count',
'redpanda.iceberg.rest_client_total_inbound_bytes.count',
'redpanda.iceberg.rest_client_total_outbound_bytes.count',
'redpanda.iceberg.rest_client_total_puts.count',
'redpanda.iceberg.rest_client_total_requests.count',
'redpanda.iceberg.translation_decompressed_bytes_processed.count',
'redpanda.iceberg.translation_dlq_files_created.count',
'redpanda.iceberg.translation_files_created.count',
'redpanda.iceberg.translation_invalid_records.count',
'redpanda.iceberg.translation_parquet_bytes_added.count',
'redpanda.iceberg.translation_parquet_rows_added.count',
'redpanda.iceberg.translation_raw_bytes_processed.count',
'redpanda.iceberg.translation_translations_finished.count',
],
'redpanda.transform': [
'redpanda.transform.execution_errors.count',
'redpanda.transform.execution_latency_sec.count',
'redpanda.transform.execution_latency_sec.bucket',
'redpanda.transform.execution_latency_sec.sum',
'redpanda.transform.failures.count',
'redpanda.transform.processor_lag.count',
'redpanda.transform.read_bytes.count',
'redpanda.transform.state',
'redpanda.transform.write_bytes.count',
],
'redpanda.wasm': [
'redpanda.wasm.binary_executable_memory_usage',
'redpanda.wasm.engine_cpu_seconds.count',
'redpanda.wasm.engine_max_memory',
'redpanda.wasm.engine_memory_usage',
],
'redpanda.tls': [
'redpanda.tls.certificate_expires_at_timestamp_seconds',
'redpanda.tls.certificate_serial',
'redpanda.tls.certificate_valid',
'redpanda.tls.loaded_at_timestamp_seconds',
'redpanda.tls.truststore_expires_at_timestamp_seconds',
],
'redpanda.authorization': [
'redpanda.authorization.result.count',
],
'redpanda.security': [
'redpanda.security.audit_errors.count',
'redpanda.security.audit_last_event_timestamp_seconds.count',
],
}
# fmt: on

INSTANCE_DEFAULT_GROUPS = [
'redpanda.application',
'redpanda.authorization',
'redpanda.cluster',
'redpanda.io_queue',
'redpanda.kafka',
Expand All @@ -155,16 +294,22 @@
'redpanda.raft',
'redpanda.reactor',
'redpanda.rpc',
'redpanda.security',
'redpanda.storage',
'redpanda.tls',
]

INSTANCE_ADDITIONAL_GROUPS = [
'redpanda.cloud',
'redpanda.controller',
'redpanda.debug_bundle',
'redpanda.node_status',
'redpanda.pandaproxy',
'redpanda.scheduler',
'redpanda.schemaregistry',
'redpanda.iceberg',
'redpanda.transform',
'redpanda.wasm',
]


Expand Down
Loading
Loading