[data.dashboard] Fix broken Ray Data per node metrics because of unsupported operator filter #57970
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request addresses an issue with broken Ray Data per-node metrics on the dashboard by removing an unsupported operator filter from the corresponding Prometheus queries. The changes are applied to the OUTPUT_BYTES_BY_NODE_PANEL, BLOCKS_BY_NODE_PANEL, and TASK_THROUGHPUT_BY_NODE_PANEL definitions. Since the underlying per-node metrics do not have an operator label, removing this filter is the correct fix to ensure data is displayed on these panels. The changes are accurate and well-targeted. I have no further suggestions.
iamjustinhsu
approved these changes
Oct 21, 2025
…pported operator filter Signed-off-by: cong.qian <[email protected]>
coqian
force-pushed
the
coqian/fix-per-node-metrics
branch
from
2023495 to
f3b637f
Compare
ray-gardener
bot
added
data
alanwguo
approved these changes
Oct 22, 2025
bveeramani
requested changes
Oct 27, 2025
Member
bveeramani
left a comment
bveeramani
left a comment
There was a problem hiding this comment.
- Rather than removing the
operatorfilter, should we addoperatoras a tag to the per-node gauges? - Are these metrics still useful if they're not disambiguated by operator?
Contributor
Author
@bveeramani According to the original PR, these three metrics are designed to be aggregated at the per node level instead of operator level. And they are useful to provide the visualization segmented by node cc @omatthew98 |
Member
Huh, okay. If this is consistent with the original intent, SGTM |
bveeramani
approved these changes
Oct 27, 2025
landscapepainter
pushed a commit
to landscapepainter/ray
that referenced
this pull request
Nov 17, 2025
…pported operator filter (ray-project#57970) ## Description The per node metrics at OSS Ray Data dashboard are not displayed as expected. Because of this code change ray-project#55495, the following three metrics were added a filter for `operator`, which is [not supported](https://github.com/ray-project/ray/blob/e51f8039bc6992d37834bcff109a3d340e78fcde/python/ray/data/_internal/stats.py#L448) by per node metrics, and causes empty result. ray_data_num_tasks_finished_per_node ray_data_bytes_outputs_of_finished_tasks_per_node ray_data_blocks_outputs_of_finished_tasks_per_node Signed-off-by: cong.qian <[email protected]>
Aydin-ab
pushed a commit
to Aydin-ab/ray-aydin
that referenced
this pull request
Nov 19, 2025
…pported operator filter (ray-project#57970) ## Description The per node metrics at OSS Ray Data dashboard are not displayed as expected. Because of this code change ray-project#55495, the following three metrics were added a filter for `operator`, which is [not supported](https://github.com/ray-project/ray/blob/e51f8039bc6992d37834bcff109a3d340e78fcde/python/ray/data/_internal/stats.py#L448) by per node metrics, and causes empty result. ray_data_num_tasks_finished_per_node ray_data_bytes_outputs_of_finished_tasks_per_node ray_data_blocks_outputs_of_finished_tasks_per_node Signed-off-by: cong.qian <[email protected]> Signed-off-by: Aydin Abiar <[email protected]>
Future-Outlier
pushed a commit
to Future-Outlier/ray
that referenced
this pull request
Dec 7, 2025
…pported operator filter (ray-project#57970) ## Description The per node metrics at OSS Ray Data dashboard are not displayed as expected. Because of this code change ray-project#55495, the following three metrics were added a filter for `operator`, which is [not supported](https://github.com/ray-project/ray/blob/e51f8039bc6992d37834bcff109a3d340e78fcde/python/ray/data/_internal/stats.py#L448) by per node metrics, and causes empty result. ray_data_num_tasks_finished_per_node ray_data_bytes_outputs_of_finished_tasks_per_node ray_data_blocks_outputs_of_finished_tasks_per_node Signed-off-by: cong.qian <[email protected]> Signed-off-by: Future-Outlier <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The per node metrics at OSS Ray Data dashboard are not displayed as expected.
Because of this code change #55495, the following three metrics were added a filter for
operator, which is not supported by per node metrics, and causes empty result.ray_data_num_tasks_finished_per_node
ray_data_bytes_outputs_of_finished_tasks_per_node
ray_data_blocks_outputs_of_finished_tasks_per_node