Queue: wrap up e2e workflow

[shcheklein]

Followup after #3091

### Tasks
- [ ] Watch and keep updating plots
- [x] When something runs in background - can we allow running an experiment in workspace?
- [x] Attach and follow standard output / error
- [ ] Failed / stoped - add resume and / or run again action
- [x] Add to a queue action is very slow - can we add a progress bar and notification? (fixed by https://github.com/iterative/vscode-dvc/pull/3815 without adding a notification)

See #3091 (comment)

[shcheklein]

I've updated the ticket.

[mattseddon]

Watch and keep updating plots

Depends on treeverse/dvc#8478. Unactionable until that is closed.

[mattseddon]

@shcheklein can you please reorder the checkboxes in order of priority? I will add some comments.

[shcheklein]

@mattseddon done (also, trying a new beta feature - tasklists).

[mattseddon]

When something runs in background - can we allow running an experiment in workspace?

We need to update the extension's package.json and the experiments table to make this happen. It should be achievable to enable. However, I have manually tested this and the experience feels very unstable.

Attach and follow standard output / error

It can be done with dvc queue logs <exp-id> -f but will need a sizeable amount of wiring built around it. We would need to use the PseudoTerminal and build a wrapper around it that looks a lot like the DvcRunner. Should be enabled from the right-click context menu in the table which will bring its own set of challenges (can it be run twice for the same experiment, should it be disabled, under what circumstances, etc).

Failed / stoped - add resume and / or run again action

I don't think this is currently possible. Maybe we could do something with a combination of apply, re-queue and start the queue but I feel like it is unlikely that it will work as expected. Feels like this needs a FR against DVC.

Add to a queue action is very slow - can we add a progress bar and notification?

What would the progress notification show and when would it update? From the data we get back, it would go from being empty to being full and that would be it (we have no information about intermediate steps). My suggestion for this would be to raise an issue in DVC to make the command more performant.

[mattseddon]

I don't think we can mark this as done until treeverse/dvc#8763 is resolved.

[shcheklein]

Thanks @mattseddon, for giving more color / details to the list.

However, I have manually tested this and the experience feels very unstable.

Could you please add more color/details to this?

It can be done with dvc queue logs -f but will need a sizeable amount of wiring built around it. We would need to use the PseudoTerminal and build a wrapper around it that looks a lot like the DvcRunner. Should be enabled from the right-click context menu in the table which will bring its own set of challenges (can it be run twice for the same experiment, should it be disabled, under what circumstances, etc).

can we cut the scope? Show some tooltip with static output when you show it , dump logs as a file and open it, etc? Clearly ppl will have to run it again to get the most recent result. But at least it something. It's way better compared to nothing.

Failed / stoped - add resume and / or run again action
I don't think this is currently possible. Maybe we could do something with a combination of apply, re-queue and start the > queue but I feel like it is unlikely that it will work as expected. Feels like this needs a FR against DVC.

@dberenbaum could you please chime in? What's your take on being able to resume / queue again a failed / killed experiment(s)?

My suggestion for this would be to raise an issue in DVC to make the command more performant.

It's not possible I think. We can get it only to a certain level. I see that it's problematic w/o DVC support. We need to start thinking about improving this though - for data commands, for queue, for all commands that take long time to run.