[Bug] Sometimes the worker node logs in the ray dashboard are empty

[chhabrakadabra]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Dashboard

What happened + What you expected to happen

The "logs" overview for worker nodes is often empty. See this screenshot for what that looks like:

Note that when I open the chrome inspector, I don't see any logs. When I look at the network tab and click on the logs link, I see a GET request to /dashboard/node_logs?ip=10.15.87.61&pid=, with this response: {"result": true, "msg": "Fetched logs.", "data": {"logs": {}}}.

The actual response from the backend is empty. But the link to the logs in the screenshot shows that something else thinks that there are 44 lines of logs.

Versions / Dependencies

• ray[default,tune]==1.9.1
• python==3.7.9
• ubuntu 18.04

Reproduction script

I imagine that spinning up a ray cluster with a dashboard on this version and doing anything that generates log will reproduce the issue.

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[rkooo567]

@edoakes I will mark dashboard issues with dashboard label

[skabbit]

Got the same issue on

ray==1.12.0
python==3.9.8

[moured]

Got the same issue. Python 3.9, the latest ray version (today). This Log image should show 20 requests but nothing is there :(

[scottsun94]

@chhabrakadabra @skabbit @moured Thanks for reporting the issue!
In 2.0, we have changed the previous experimental dashboard to the default dashboard. If you click on the logs button in the node table, it will send you to the logs page.

Does it solve your issues?