The cleanup script doesn't include any actual cleanup logic

[dkmiller]

All it does is "iterate through" all webservices and compute targets.

Recommendation: it should shut down / scale down all actively running compute.

[lostmygithubaccount]

closing, linked wrong issue # in the PR

[lostmygithubaccount]

whoops my bad

[lostmygithubaccount]

for this issue, one problem is all of the compute types return different things:

[lostmygithubaccount]

so the example cleanup script does something like this:

# delete some compute targets
for compute_target in ws.compute_targets:
    if ws.compute_targets[compute_target].get_status() in ["Failed", "Canceled", None]:
        try:
            ws.compute_targets[compute_target].delete()
        except:
            pass
    elif (
        "dask-ct" in compute_target
        and ws.compute_targets[compute_target]
        .get_status()
        .serialize()["provisioningState"]
        in ["Succeeded"]
        and len(ws.compute_targets[compute_target].list_nodes()) == 0
    ):
        ws.compute_targets[compute_target].delete()

should we maybe check if any clusters have nodes > 1 and do something?

I'm a little afraid to have any real logic in this cleanup script given a user might not notice and unintentionally have their compute deleted/runs cancelled

[dkmiller]

I'd suggest checking for clusters with ≥ 1 nodes that aren't running (to make sure we don't kill active runs) and then forcibly scale them down to zero nodes.