fix controller job cache not sync latest version issue

[wpeng102]

Signed-off-by: wpeng102 [email protected]

Resolve issue: #2160

What happened:

vcjob phase can get stuck in "Pending" or "Starting" even after all pods have completed.
Detail refer #2160

Root Cause:

When controller update job status, Kopf add annotation for this job.

Consider this situation:

volcano/pkg/controllers/job/job_controller_actions.go

Lines 462 to 468 in ccda758

	newJob, err := cc.vcClient.BatchV1alpha1().Jobs(job.Namespace).UpdateStatus(context.TODO(), job, metav1.UpdateOptions{})
	if err != nil {
	klog.Errorf("Failed to update status of Job %v/%v: %v",
	job.Namespace, job.Name, err)
	return err
	}
	if e := cc.cache.Update(newJob); e != nil {

volcano/pkg/controllers/job/job_controller_handler.go

Line 93 in ccda758

if err := cc.cache.Update(newJob); err != nil {

thread1 at L462, get the job with version_1.
thread2 at L93, get Kopf add annotation event and put version_2 job into cache.
thread1 at L468, put version_1 job into cache again.

When next call UpdateStatus will use the old version_1 job, then meet 2160 issue.

Manual reproduce steps:

1. add time.Sleep(time.Second*20) before

   volcano/pkg/controllers/job/job_controller_actions.go

   Line 468 in ccda758

   if e := cc.cache.Update(newJob); e != nil {

2. submit vcjob
3. during controller sleep, add annotation for this job manually.

Solution:

In cache.update need check if the job resourceversion is bigger than the old one.

After this fix, when trigger the above case the controller log will print:

E0414 08:12:30.419699       1 job_controller_actions.go:307] SyncJob - Failed to update Job default/job-38 in cache:  job <default/job-38> has too old resource version: 6358346 (6358347)

Then controller will requeue this request.

[william-wang]

If the newResourceversion is less than odResourceversion, the cache keep the same as it was, giving a warning log and returning nil seems more reasonable.
fox example, in the killJob func, if cc.cache.Update(newJob) return error in this case, the podgroup will not be cleaned in system.

[wpeng102]

If just log a warning log and return nil will not trigger request rerun, Actually, processNextReq in job_controller will check st.Execute(action) return result, if has error, it will re-queue this request and next will get the latest job from cache.
killJob func also follow the above rule.

[Thor-wl]

/lgtm

[Thor-wl]

/approve

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Thor-wl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/controllers/OWNERS [Thor-wl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment