An Indexed Job mode that allows every index to execute

[ahg-g]

What would you like to be added?

A mode of operation for Jobs with .spec.completionMode="Indexed that allows every index to execute.

Currently this is not possible because when a job reaches its .spec.backoffLimit, active pods will be deleted; moreover, the job is declared failed and so no new pods are created for the indices that didn't execute yet (happens more often when parallelism < completions).

I can think of two open issues:

1. How to decide when to stop retrying a failed index. One approach is to consider backofflimit at the index level, this will be challenging to track per index in the job status, but one solution is to have the backofflimit with min semantics: each index is guaranteed to at least backofflimit retries, we track that in the job-controller memory for each index, and in status we only track which indexes reached the limit as a bitmap.
2. Job failure status: in the simplest case we could just declare the job failed if at least one index failed, but we could also introduce an API to allow users to tune that (perhaps based on a percentage or a min number of indexes).

Related comment to this issue: #109131 (comment)

Why is this needed?

There are cases where the indexes represent independent operations, and so it is desired to continue and execute all of them before declaring the job complete.

[k8s-ci-robot]

@ahg-g: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ahg-g]

/sig apps
/wg batch

[alculquicondor]

There is one more consideration:
For Indexed Jobs we limited the maximum value for parallelism. This is important so that the .status.completedIndexes doesn't go over a threshold. But the limit only works with the assumption that we always create the small indexes first. So if index 0 fails, we recreate a pod for it before creating a pod for index parallelism.
This assumption is now changing. So for this new mode, we might need to limit the maximum value for completions.

In that line, I think a way to solve the main problem of this issue is to introduce an indexCreationPolicy.

• SmallestFirst (current): create the smallest indexes before creating new indexes.
• NextIndex: Create the next index that hasn't been tried before. I think we might need to add the .status.failedIndexes in order to be able to implement this.

But then we also need a policy for what to do when backoffLimit is reached. The current behavior is to abort all running pods. We need one for continuing. Not sure if these 2 policies can fit into a single field. Naming becomes very hard at that point.

[alculquicondor]

I don't think the "minimum" interpretation of a backoff limit is ideal. We will certainly have confused users about extra pods being created when there is a control plane upgrade, or kcm simply crashes.

[ahg-g]

I don't think the "minimum" interpretation of a backoff limit is ideal. We will certainly have confused users about extra pods being created when there is a control plane upgrade, or kcm simply crashes.

It is not ideal, but I think it is a solution worth considering given the complexity of tracking the exact number of retries per index. Confusion can be addressed by having the semantics clarified in the name, MinRetries for example.

btw, does the current BackoffLimit guarantee the exact number of retires? also, when restartPolicy = "OnFailure", does each kubelet restart count against the BackoffLimit?

[ahg-g]

This assumption is now changing. So for this new mode, we might need to limit the maximum value for completions.

A 1 million limit, which is ~10x number of workload pods supported now is equivalent to 122KB max of storage? Should be acceptable, no?

In that line, I think a way to solve the main problem of this issue is to introduce an indexCreationPolicy.

Not sure I understand why we need this. Currently we consider an index done and move to the next if it succeeded. If we have BackoffLimit per index, then the policy changes to "if the index succeeds or it reaches its backofflimit", the rest should be the same.

[alculquicondor]

It is not ideal, but I think it is a solution worth considering given the complexity of tracking the exact number of retries per index. Confusion can be addressed by having the semantics clarified in the name, MinRetries for example.

Other than confusion, users might want stronger guarantees, as pods=cost.

btw, does the current BackoffLimit guarantee the exact number of retires? also, when restartPolicy = "OnFailure", does each kubelet restart count against the BackoffLimit?

I don't think so, as the retries are stored in the status. However, if there is a new pod, then yes, the entire old pod only counts as one failure.

A 1 million limit, which is ~10x number of workload pods supported now is equivalent to 122KB max of storage? Should be acceptable, no?

We should probably use the same limit that we set for parallelism (10^5). I don't see why we should be inconsistent.

Not sure I understand why we need this. Currently we consider an index done and move to the next if it succeeded. If we have BackoffLimit per index, then the policy changes to "if the index succeeds or it reaches its backofflimit", the reset should be the same.

I was trying to go with the assumption that we can't implement a backoffLimit per index.

[jensentanlo]

Thanks for creating this issue, I missed the notification so apologies for the delay.

I've been thinking about this since our previous discussion and I remember that you mentioned about specifying which container exit codes are considered as failures/successes. Is there a separate issue somewhere with the discussion of that proposal and why that was rejected/deferred? I think that would cover most if not all of the edge cases.