[data] reset cpu + gpu metrics on executor shutdown and updating task submission/block generation metrics #57246

iamjustinhsu · 2025-10-06T23:51:43Z

Why are these changes needed?

On executor shutdown, the metrics persist even after execution. The plan is to reset on streaming_executor.shutdown. This PR also includes 2 potential drive-by fixes for metric calculation

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run pre-commit jobs to lint the changes in this PR. (pre-commit setup)
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: iamjustinhsu <[email protected]>

iamjustinhsu · 2025-10-06T23:51:58Z

python/ray/dashboard/modules/metrics/dashboards/data_dashboard_panels.py

    targets=[
        Target(
-            expr='sum(ray_data_block_generation_time{{{global_filters}, operator=~"$Operator"}}) by (dataset, operator)',
+            expr='increase(ray_data_block_generation_time{{{global_filters}, operator=~"$Operator"}}[5m]) / increase(ray_data_num_task_outputs_generated{{{global_filters}, operator=~"$Operator"}}[5m])',


W/O PR: shows total sum of block generation time (meaningless)
W/ PR: shows average block generation time over 5min period

iamjustinhsu · 2025-10-06T23:52:04Z

python/ray/dashboard/modules/metrics/dashboards/data_dashboard_panels.py

    targets=[
        Target(
-            expr='sum(ray_data_task_submission_backpressure_time{{{global_filters}, operator=~"$Operator"}}) by (dataset, operator)',
+            expr='increase(ray_data_task_submission_backpressure_time{{{global_filters}, operator=~"$Operator"}}[5m]) / increase(ray_data_num_tasks_submitted{{{global_filters}, operator=~"$Operator"}}[5m])',


W/O PR: shows total sum of submitted tasks (could be meaningful)
W/ PR: shows current # of submitted tasks (I find this more meaningful)

Signed-off-by: iamjustinhsu <[email protected]>

…/flush-metrics

goutamvenkat-anyscale · 2025-10-09T22:21:56Z

python/ray/data/_internal/execution/streaming_executor.py

                include_parent=False
            )
-            # Reset the scheduling loop duration gauge.
-            self._sched_loop_duration_s.set(0, tags={"dataset": self._dataset_id})


is this meant to be nuked?

yes, the update_metrics calls it

bveeramani · 2025-10-21T19:05:33Z

python/ray/data/_internal/execution/resource_manager.py

+        assert isinstance(self.op_resource_allocator, ReservationOpResourceAllocator)
+        for op in self._op_usages:
+            self._op_usages[op] = ExecutionResources.zero()
+            self.op_resource_allocator._op_budgets[op] = ExecutionResources.zero()
+            self.op_resource_allocator._output_budgets[op] = 0


This is brittle and breaks abstraction barriers. If we change the implementation of ReservationOpResourceAllocator or change the allocator altogether, this could break

good shout, updated

bveeramani · 2025-10-21T19:06:26Z

python/ray/data/_internal/execution/streaming_executor.py

-            # Reset the scheduling loop duration gauge.
-            self._sched_loop_duration_s.set(0, tags={"dataset": self._dataset_id})
+            # Reset the scheduling loop duration gauge + resource manager budgets/usages.
+            self._resource_manager.clear_usages_and_budget()


@iamjustinhsu if we call the regular update_usages here, does that clear the budget and usages? If not, why?

It does not entirely, because budget metrics are done outside of resource manager. On shutdown, the budgets, ie, it will show a line of the last updated budget/usage, which will non-zero

ok fixed it with latest commits. The reason budget wasn't resetting was due to how we clear the budgets for each operator when we call update_usages. See #57246 (comment)

Signed-off-by: iamjustinhsu <[email protected]>

python/ray/data/_internal/execution/resource_manager.py

Signed-off-by: iamjustinhsu <[email protected]>

bveeramani

Nice

…/flush-metrics

… submission/block generation metrics (ray-project#57246)   ## Why are these changes needed? On executor shutdown, the metrics persist even after execution. The plan is to reset on streaming_executor.shutdown. This PR also includes 2 potential drive-by fixes for metric calculation  ## Related issue number  ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: xgui <[email protected]>

… submission/block generation metrics (ray-project#57246)   ## Why are these changes needed? On executor shutdown, the metrics persist even after execution. The plan is to reset on streaming_executor.shutdown. This PR also includes 2 potential drive-by fixes for metric calculation  ## Related issue number  ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: iamjustinhsu <[email protected]>

… submission/block generation metrics (ray-project#57246)   ## Why are these changes needed? On executor shutdown, the metrics persist even after execution. The plan is to reset on streaming_executor.shutdown. This PR also includes 2 potential drive-by fixes for metric calculation  ## Related issue number  ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: Aydin Abiar <[email protected]>

… submission/block generation metrics (ray-project#57246)   ## Why are these changes needed? On executor shutdown, the metrics persist even after execution. The plan is to reset on streaming_executor.shutdown. This PR also includes 2 potential drive-by fixes for metric calculation  ## Related issue number  ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: Future-Outlier <[email protected]>

[data] reset metrics on executor shutdown

4362718

Signed-off-by: iamjustinhsu <[email protected]>

iamjustinhsu commented Oct 6, 2025

View reviewed changes

iamjustinhsu added 3 commits October 6, 2025 17:33

lint

c0d459b

Signed-off-by: iamjustinhsu <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into jhsu…

780dee1

…/flush-metrics

Merge branch 'master' of https://github.com/ray-project/ray into jhsu…

1216e54

…/flush-metrics

iamjustinhsu changed the title ~~[data] reset metrics on executor shutdown~~ [data] reset cpu + gpu metrics on executor shutdown Oct 9, 2025

goutamvenkat-anyscale reviewed Oct 9, 2025

View reviewed changes

iamjustinhsu marked this pull request as ready for review October 9, 2025 22:48

iamjustinhsu requested a review from a team as a code owner October 9, 2025 22:48

This comment was marked as outdated.

Sign in to view

ray-gardener bot added data Ray Data-related issues observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling labels Oct 10, 2025

iamjustinhsu changed the title ~~[data] reset cpu + gpu metrics on executor shutdown~~ [data] reset cpu + gpu metrics on executor shutdown and updating task submission/block generation metrics Oct 20, 2025

bveeramani reviewed Oct 21, 2025

View reviewed changes

iamjustinhsu added 2 commits October 21, 2025 14:09

zero out budget without clearing

0d4339f

Signed-off-by: iamjustinhsu <[email protected]>

lint

c71130e

Signed-off-by: iamjustinhsu <[email protected]>

This comment was marked as outdated.

Sign in to view

iamjustinhsu added 2 commits October 21, 2025 14:19

fix

b0672a8

Signed-off-by: iamjustinhsu <[email protected]>

lint

ada3c90

Signed-off-by: iamjustinhsu <[email protected]>

iamjustinhsu commented Oct 21, 2025

View reviewed changes

python/ray/data/_internal/execution/resource_manager.py Outdated Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

iamjustinhsu added 5 commits October 21, 2025 15:51

clearer?

8f12426

Signed-off-by: iamjustinhsu <[email protected]>

clearer? pt2

a95b892

Signed-off-by: iamjustinhsu <[email protected]>

ugh

de78ced

Signed-off-by: iamjustinhsu <[email protected]>

grammar

0aa2049

Signed-off-by: iamjustinhsu <[email protected]>

guh

9a25054

Signed-off-by: iamjustinhsu <[email protected]>

This comment was marked as outdated.

Sign in to view

use None

5838314

Signed-off-by: iamjustinhsu <[email protected]>

This comment was marked as outdated.

Sign in to view

try this

24f87eb

Signed-off-by: iamjustinhsu <[email protected]>

bveeramani approved these changes Oct 23, 2025

View reviewed changes

iamjustinhsu added the go add ONLY when ready to merge, run all tests label Oct 23, 2025

Merge branch 'master' of https://github.com/ray-project/ray into jhsu…

003ae5b

…/flush-metrics

alexeykudinkin merged commit a6ccbb3 into ray-project:master Oct 24, 2025
6 checks passed

iamjustinhsu deleted the jhsu/flush-metrics branch October 24, 2025 19:00

[data] reset cpu + gpu metrics on executor shutdown and updating task submission/block generation metrics #57246

[data] reset cpu + gpu metrics on executor shutdown and updating task submission/block generation metrics #57246

Uh oh!

Conversation

iamjustinhsu commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

iamjustinhsu Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goutamvenkat-anyscale Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

bveeramani Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

bveeramani Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

bveeramani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iamjustinhsu commented Oct 6, 2025 •

edited

Loading

iamjustinhsu Oct 6, 2025 •

edited

Loading

iamjustinhsu Oct 6, 2025 •

edited

Loading

iamjustinhsu Oct 21, 2025 •

edited

Loading

iamjustinhsu Oct 21, 2025 •

edited

Loading