[data] Add dataset/operator state, progress, total metrics #50770

omatthew98 · 2025-02-20T20:06:46Z

Add various metrics that are captured in the progress bar but are not captured in the prometheus metrics emitted.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

can-anyscale

w00h00

python/ray/data/_internal/stats.py

can-anyscale · 2025-02-20T20:18:17Z

python/ray/data/_internal/stats.py

@@ -266,6 +266,44 @@ def __init__(self, max_stats=1000):
            tag_keys=iter_tag_keys,
        )

+        # === Dataset and Operator Metadata Metrics ===
+        dataset_tags = ("dataset",)


nit,

dataset_tags = ("dataset", "jobid", "total", "starttime") (these are all metadata that never changes)

full schema in https://github.com/anyscale/rayturbo/pull/1446/files#diff-dc8f3eb3e3b8d8b0b62c1160389cad945623fd82e78ec1d2d54a45fe26902320

not sure if i understand the meaning of dataset_tags correctly, but i just mean to also store jobid, total and startime as column/label in prometheus

Total might change afaik so leaving that off. I'm not sure if it makes sense to have jobid or starttime as tags here as that might increase the cardinality of our metrics tags. Are those necessary for the data dashboard?

Yess, startime is a column in data dashboard and jobid is an important link to render job status, etc. I believe the cardinality size is affected by the number of combination in the label values. In this case, because starttime and jobid values are singleton to dataset, it doesn't affect cardinality size, if that makes sense.

if you believe chatgpt ;)

@omatthew98 can we discuss/do this before merging, otherwise I won't be able to generate enough data for the dashboard with just metrics

Let me discuss with the team, adding jobid seems reasonable here, dataset start time as a tag feels like a bit of an antipattern.

FWIW these are the two things suggested by ChatGPT (neither which feel particularly elegant).

For static metadata like a job ID or start time, it's common practice not to create separate gauges for each piece of information. Instead, you can adopt one of these approaches:

Info Metric Pattern:
Create a single gauge (often set to a constant value like 1) with labels representing your static metadata. For instance, you might have a metric called job_info with labels such as jobid and start_time. This pattern is widely used (e.g., many exporters expose a build_info metric) and keeps your metric space clean.

Const Labels:
When initializing a metric that is associated with a particular job, you can add the metadata as constant labels. This is useful if the metadata is truly static for the lifetime of that metric instance and you don't expect many different values that could lead to high cardinality.

Both methods allow you to expose useful metadata in Prometheus without having to create multiple separate gauges. Just be cautious with label cardinality—if your metadata values vary too widely, they might negatively impact your Prometheus queries and storage.

These tags don't make sense in Ray Turbo

Jobid: we don't know where RT will be running so jobid is not well-defined

Starttime: is unbounded hence can't be a tag

Total: i don't even understand what this could be

Const Labels:
When initializing a metric that is associated with a particular job, you can add the metadata as constant labels. This is useful if the metadata is truly static for the lifetime of that metric instance and you don't expect many different values that could lead to high cardinality.

This is what I'm suggesting basically ;). If they are constants then there should be no downside in storing this in prometheus. We actually already have 20+ constant labels (cluster related information) for every metric in prometheus.

@alexeykudinkin: if I understand your comments correctly - don't think about them as tags; labels are just constant columns in prometheus (unless these tags have different semantic in dataset point of views)

Info Metric Pattern:
Create a single gauge (often set to a constant value like 1) with labels representing your static metadata. For instance, you might have a metric called job_info with labels such as jobid and start_time. This pattern is widely used (e.g., many exporters expose a build_info metric) and keeps your metric space clean.

This is also fine except that it's more expensive (additional metrics vs. a constant column like the other solutions)

python/ray/data/_internal/stats.py

can-anyscale · 2025-03-04T21:11:57Z

python/ray/data/_internal/stats.py

            tag_keys=dataset_tags,
        )
        self.dataset_state = Gauge(
-            "ray_data_dataset_state",
+            "data_dataset_state",
            description=f"State of dataset ({', '.join([f'{s.value}={s.name}' for s in DatasetState])})",
            tag_keys=("dataset",),


nit: dataset_tags

raulchen · 2025-03-04T21:42:35Z

python/ray/data/_internal/stats.py

+            tag_keys=operator_tags,
+        )
+        self.operator_total = Gauge(
+            "data_operator_total",


Suggested change

"data_operator_total",

"data_operator_estimated_total_blocks",

This is the estimated total number of blocks right?

Same for other "total" metrics. let's add "estimated" to avoid confusion.

raulchen · 2025-03-04T21:50:48Z

python/ray/data/_internal/stats.py

+        operator_tags = ("dataset", "operator")
+        self.operator_progress = Gauge(
+            "data_operator_progress",
+            description="Progress of operator execution",


Is this num rows or blocks? I think we need both

This is actually the number of completed tasks

ray/python/ray/data/_internal/execution/streaming_executor.py

Line 410 in 6cf3c72

"progress": last_state.num_completed_tasks,

. I think we already have the output of the dataset in rows and blocks here and here.

Can we make the name more explicit?

Signed-off-by: Matthew Owen <[email protected]>

can-anyscale

can-anyscale · 2025-03-06T16:27:05Z

python/ray/data/_internal/stats.py

        self.datasets[dataset_tag].update(state)
+        job_id = self.datasets[dataset_tag].get("job_id", None)


nit: job_id = self.datasets[dataset_tag].get("job_id") (get returns None by default)

also not sure if prometheus handle null/none gracefully (prometheus/client_java#315) so maybe gives it some dummy default value instead of None.

alexeykudinkin · 2025-03-06T18:16:20Z

python/ray/data/_internal/stats.py

+        self.data_dataset_progress = Gauge(
+            "data_dataset_progress",
+            description="Progress of dataset execution",
+            tag_keys=dataset_tags,
+        )


What are these?

Explanation here: #50770 (comment)

I will make more explicit though like @raulchen suggested.

Signed-off-by: Matthew Owen <[email protected]>

…ct#50770) Add various metrics that are captured in the progress bar but are not captured in the prometheus metrics emitted. --------- Signed-off-by: Matthew Owen <[email protected]>

…ct#50770) Add various metrics that are captured in the progress bar but are not captured in the prometheus metrics emitted. --------- Signed-off-by: Matthew Owen <[email protected]> Signed-off-by: Jay Chia <[email protected]>

…ct#50770) Add various metrics that are captured in the progress bar but are not captured in the prometheus metrics emitted. --------- Signed-off-by: Matthew Owen <[email protected]>

omatthew98 requested a review from a team as a code owner February 20, 2025 20:06

omatthew98 assigned can-anyscale Feb 20, 2025

can-anyscale reviewed Feb 20, 2025

View reviewed changes

can-anyscale reviewed Feb 28, 2025

View reviewed changes

python/ray/data/_internal/stats.py Outdated Show resolved Hide resolved

can-anyscale self-requested a review March 3, 2025 21:39

can-anyscale reviewed Mar 4, 2025

View reviewed changes

raulchen reviewed Mar 4, 2025

View reviewed changes

raulchen approved these changes Mar 4, 2025

View reviewed changes

omatthew98 added the go add ONLY when ready to merge, run all tests label Mar 5, 2025

omatthew98 added 8 commits March 5, 2025 22:34

first stab at adding prom metrics for progress / state

957a9da

Signed-off-by: Matthew Owen <[email protected]>

pr feedback, introduce enum

92c02fc

Signed-off-by: Matthew Owen <[email protected]>

remove ray prefix to match existing metric names

2cb138e

Signed-off-by: Matthew Owen <[email protected]>

adding in total_rows

61789af

Signed-off-by: Matthew Owen <[email protected]>

respond to pr feedback

5706b1b

Signed-off-by: Matthew Owen <[email protected]>

fix test_data_head

701a146

Signed-off-by: Matthew Owen <[email protected]>

robocop

a13a221

Signed-off-by: Matthew Owen <[email protected]>

adding in tags for jobid and start time

c70e5c9

Signed-off-by: Matthew Owen <[email protected]>

omatthew98 force-pushed the mowen/add-more-data-prom-metrics branch from ebb795e to c70e5c9 Compare March 6, 2025 06:34

appease

e7a9366

Signed-off-by: Matthew Owen <[email protected]>

can-anyscale approved these changes Mar 6, 2025

View reviewed changes

raulchen approved these changes Mar 6, 2025

View reviewed changes

alexeykudinkin reviewed Mar 6, 2025

View reviewed changes

removing dataset and operator progress

d509f5a

Signed-off-by: Matthew Owen <[email protected]>

raulchen merged commit 0cf2a4b into ray-project:master Mar 7, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[data] Add dataset/operator state, progress, total metrics #50770

[data] Add dataset/operator state, progress, total metrics #50770

omatthew98 commented Feb 20, 2025

can-anyscale left a comment

can-anyscale Feb 20, 2025

can-anyscale Feb 20, 2025

omatthew98 Mar 3, 2025

can-anyscale Mar 3, 2025

can-anyscale Mar 3, 2025

can-anyscale Mar 4, 2025

omatthew98 Mar 5, 2025

alexeykudinkin Mar 5, 2025

can-anyscale Mar 5, 2025

can-anyscale Mar 5, 2025

can-anyscale Mar 4, 2025

raulchen Mar 4, 2025

raulchen Mar 4, 2025

raulchen Mar 4, 2025

omatthew98 Mar 4, 2025

raulchen Mar 6, 2025

can-anyscale left a comment

can-anyscale Mar 6, 2025

alexeykudinkin Mar 6, 2025

omatthew98 Mar 6, 2025

	"data_operator_total",
	"data_operator_estimated_total_blocks",

		self.datasets[dataset_tag].update(state)
		job_id = self.datasets[dataset_tag].get("job_id", None)

[data] Add dataset/operator state, progress, total metrics #50770

[data] Add dataset/operator state, progress, total metrics #50770

Conversation

omatthew98 commented Feb 20, 2025

Why are these changes needed?

Related issue number

Checks

can-anyscale left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

can-anyscale left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment