Ray Logging and Dashboard Metrics Export to S3 with Custom Dashboard for Historical Clusters

[vara-bonthu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

The outcome we are trying to reach is to provide continuous access to the Ray Dashboard metrics and logs even after the Ray Cluster is deleted. This involves exporting all relevant data to an S3 bucket, enabling users to visualize and analyze metrics and logs with a custom dashboard. Additionally, this setup will allow the creation of a comprehensive dashboard to monitor metrics and logs across multiple clusters and jobs, maintaining a history of cluster and job activities for better tracking and analysis.

Describe the solution you would like

This feature will be useful for users who want to view the dashboard even after the Ray Cluster is deleted. Currently, the Ray Dashboard is available only during the cluster's lifetime, and you cannot access the dashboard or metrics once the cluster is deleted. With this feature, all metrics and logs can be continuously exported to an S3 bucket and visualized with a custom dashboard, allowing access to data even after the cluster is deleted. Additionally, it would be beneficial to build a custom dashboard to visualize metrics and logs for multiple clusters and jobs, maintaining a history of clusters and jobs.

Describe alternatives you have considered

Additional context

[vara-bonthu]

These issues are tracked in the AI on EKS project.
If your issue isn’t listed there, please open a new one in AI on EKS repo.