If k8s odr tasks are pending, they will never get cleaned up

[izaaklauer]

Describe the bug

If k8s tasks do not complete (or error), they will stick around forever.

Steps to Reproduce

• Enable ODR on k8s waypoint, and app and/or project polling
• Scale down the worker pool (or otherwise exhaust some resource)
• Observe as waypoint continues to spawn pending tasks forever, which will thundering herd the cluster when you add additional capacity

Expected behavior

This is because we aren't ever calling stop on k8s tasks:

waypoint/builtin/k8s/task.go

Line 138 in 283a474

// Purposely do nothing. We leverage the job TTL feature in Kube 1.19+

This is actually really nice, because without this there's no way to inspect the logs of a failed poll-based task. Maybe we can call stop after some delay period, or otherwise store the logs somewhere.

Waypoint Platform Versions
Additional version and platform information to help triage the issue if
applicable:

• Waypoint CLI Version: 0.6.1
• Waypoint Server Platform and Version: 0.6.1
• Waypoint Plugin: k8s

Additional context

Slack thread: https://hashicorp.slack.com/archives/C013QT1KG9W/p1635804287310700

[izaaklauer]

A nice solution: it would be nice if we left errored pods around so we can easily look at the logs from the k8s layer.

Two good points from @catsby:

• It might not always be important to keep the logs around, because sophisticated users will probably all pod logs shipped somewhere (like elasticsearch) that they can look at after the fact.
• We can probably leave errored pods around, because I think Kubernetes has some kind of auto-reaping policy for errored pods.