Plots from temporary experiments are first updated when finished

[ostromann]

If you start a temporary experiment by using dvc exp run --temp, the plots will show the metrics of the last completed experiment in the workspace instead of the current temporary experiment.
When the temporary experiment finishes, it disappears from the plots and needs to be re-added, showing the actual metrics then.

It would be good to see the metrics of currently running temporary experiments in realtime in the Plots section.

[mattseddon]

Hi @ostromann,

Thanks for raising the issue.

There is a limitation in that DVC does not collect metrics for experiments running outside of the workspace (unless they are checkpoint based). This is being actively worked on at treeverse/dvc#8478. We have #3178 in this repo for the subsequent work needed.

By any chance are you using DVCLive to log metrics?

[ostromann]

Thanks!
Yes, I do use DVCLive to log metrics.

[mattseddon]

Ok, as a temporary workaround, you can get live updates by connecting to Studio, switching from --temp to --queue and starting the queue with the extension. Live updates will show up in Studio for all experiments run from the queue.

Docs are here.

Hope this helps. LMK if you get stuck.

Note: I think if you start dvc exp run --temp with the environment variables STUDIO_TOKEN & STUDIO_REPO_URL included then you should still get live updates.

Plug git ls-remote --get-url into STUDIO_REPO_URL

[ostromann]

Unfortunately, I'm running experiments on a company-owned GPU server that doesn't have direct internet connection. Also, our repos are on an enterprise GitLab. As far as I understand, that rules out DVC Studio for me, right now.

We log separately to MLFlow, so that satisfies a workaround for us right now. Would be nice to only use DVC Live in the long run, though :) 👍

[shcheklein]

As far as I understand, that rules out DVC Studio for me, right now

There is an on-prem option for Studio (it's paid, but just in case your org can consider this). In no way saying we are not fixing this, we are working on this, it's prioritized.

[mattseddon]

@dberenbaum this issue is why we want to prioritise live plot updates for experiments running in the queue/temp directories. Collecting data for the separate queue/temp revisions would be problematic for the reasons mentioned in #3676. We would have to re-implement the "merging" logic, from DVC in the extension, make separate CLI calls, cache data etc. This would be a significant effort in the extension and IMO less valuable than implementing the ability to call plots diff with these revisions in DVC. I'd be happy to implement it on the DVC side if we decide it would be the best way forwards.

LMK what you think, 🙏🏻.

Edit: rereading the above I see that I already mentioned treeverse/dvc#8478 I'll make a similar comment in that issue as that is probably a better place to discuss.

[dberenbaum]

Can we decide how it should work in #3676?