simplify reconciler #166

zw0610 · 2021-09-28T08:10:48Z

To support GenericJob in tf-operator, a kind of Job independent from any deep learning frameworks, this pull request introduces the following modification to reconciler.v1 package:

Simplify multiple reconciler names (by dropping redundant words like Kubeflow)
Fix PodList, ServiceList converting issue
Fix default GetReconcilerName returned value
Simplify reconciler.v1 package by moving the KubeflowReconciler to tf-operator [WIP]: add a GenericJob type and controller training-operator#1398
Simplify TestReconciler construction function

google-oss-robot · 2021-09-28T08:10:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign jeffwan after the PR has been reviewed.
You can assign the PR to them by writing /assign @jeffwan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/reconciler.v1/common/service.go

terrytangyuan · 2021-09-28T14:54:06Z

pkg/reconciler.v1/common/README.md

@@ -6,19 +6,19 @@ To use the reconciler, following methods must be overridden according to the API

 ```go
 // GetJob returns the job that matches the request
-func (r *KubeflowJobReconciler) GetJob(ctx context.Context, req ctrl.Request) (client.Object, error)
+func (r *JobReconciler) GetJob(ctx context.Context, req ctrl.Request) (client.Object, error)


What about something like Training or TrainingJob since we are always dealing with model trainings here. Job seems very generic.

TrainingJob sounds good to me. As the common library is designed for all kinds of controllers, I would suggest renaming the GenericJob to TrainingJob in the pull request I mentioned above for tf-operator (kubeflow/training-operator#1398). What do you think?

terrytangyuan · 2021-09-28T14:54:28Z

/cc @kubeflow/wg-training-leads

Jeffwan · 2021-09-29T17:06:50Z

Sorry for the delay. Let me merge #163 first before accepting any new changes. This is really annoying.

zw0610 · 2021-12-01T11:02:57Z

@Jeffwan could we move this pr forward as the revert string issue is resolved now.

terrytangyuan

LGTM
/assign @Jeffwan

Jeffwan · 2021-12-12T08:50:19Z

pkg/reconciler.v1/common/utils.go


-// GetReconcilerName returns the name of this reconciler, which is "Kubeflow Reconciler"
+// GetReconcilerName returns the name of this reconciler, which is "common-reconciler"
 func (r *ReconcilerUtil) GetReconcilerName() string {


"Kubeflow Reconciler" won't work because it's not a word?

Yes. When generating the default labels, the returned string from GetReconcilerName will be used as the value for key commonv1.OperatorNameLable (""training.kubeflow.org/operator-name"").

Jeffwan · 2021-12-12T08:52:06Z

pkg/reconciler.v1/common/interface.go

@@ -239,21 +239,3 @@ type JobInterface interface {
 	// PastActiveDeadline CAN be overridden to customize how to determine if this job has past activate deadline.
 	PastActiveDeadline(runPolicy *commonv1.RunPolicy, jobStatus *commonv1.JobStatus) bool
 }
-
-// KubeflowReconcilerInterface defines the abstract interface for a base reconciler for kubeflow jobs.
-type KubeflowReconcilerInterface interface {


What's the motivation to move KubeflowReconcilerInterface to training-operator?

When writing the test code, test_reconciler, which could be considered as one example of how the common.reconciler will be used, I found that:

Such interface will not be used for any abstraction when implementing a real reconciler

Embedding modularized interfaces in KubeflowReconcilerInterface conflicts embedding them in the real reconciler struct.

I think it make sense to me. current testing integration between common and training-operator is a headache.

Jeffwan · 2021-12-12T08:58:17Z

pkg/reconciler.v1/common/service.go

 	svcList := &corev1.ServiceList{}
 	err := r.List(ctx, svcList, client.MatchingLabels(r.GenLabels(job.GetName())))
 	if err != nil {
 		return nil, err
 	}

 	var svcs []*corev1.Service = nil
-	for _, svc := range svcList.Items {


em. what's the problem of original way?

It seems the old way will not give a new address for svc so when we take the pointer of svc, it remains the same. As a result, all Services in the svc slice are the same. Please see the screenshot below.

I see. yeah. This pointer points to the shared memory location that each of the slice items are being copied into.
All the pointers created using &svc point to the same memory location.

Current way looks good to me.

pkg/reconciler.v1/common/service.go

Jeffwan · 2021-12-12T09:00:00Z

@zw0610 sorry for late response. Please have a check on above comments

remove nil

Jeffwan · 2021-12-12T17:06:52Z

/lgtm
/hold

This change looks good to me. Let's see if anyone else have further feedbacks. /cc @kubeflow/wg-training-leads

zw0610 · 2021-12-19T07:26:54Z

Any feedback before we can merge this pr? @kubeflow/wg-training-leads

terrytangyuan · 2021-12-19T18:46:39Z

/hold cancel

terrytangyuan · 2021-12-19T18:47:03Z

/lgtm
/approve

google-oss-prow · 2021-12-19T18:47:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jeffwan, terrytangyuan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Jeffwan,terrytangyuan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-robot requested review from Jeffwan and terrytangyuan September 28, 2021 08:10

google-oss-robot added the size/L label Sep 28, 2021

terrytangyuan reviewed Sep 28, 2021

View reviewed changes

google-oss-robot requested a review from a team September 28, 2021 14:54

google-oss-robot added size/XL and removed size/L labels Sep 29, 2021

zw0610 mentioned this pull request Oct 28, 2021

WIP: Revert #135: change rtype to commonv1.ReplicaType #163

Closed

zw0610 force-pushed the master branch from 0517f2a to f320f16 Compare December 1, 2021 09:23

terrytangyuan approved these changes Dec 7, 2021

View reviewed changes

google-oss-prow bot assigned Jeffwan Dec 7, 2021

google-oss-prow bot added the approved label Dec 7, 2021

Jeffwan reviewed Dec 12, 2021

View reviewed changes

simplify reconciler

7e61759

remove nil

zw0610 force-pushed the master branch from f320f16 to 7e61759 Compare December 12, 2021 12:26

google-oss-prow bot added do-not-merge/hold lgtm labels Dec 12, 2021

Jeffwan approved these changes Dec 12, 2021

View reviewed changes

google-oss-prow bot removed the do-not-merge/hold label Dec 19, 2021

google-oss-prow bot assigned terrytangyuan Dec 19, 2021

google-oss-prow bot merged commit 4322683 into kubeflow:master Dec 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simplify reconciler #166

simplify reconciler #166

zw0610 commented Sep 28, 2021 •

edited

Loading

google-oss-robot commented Sep 28, 2021

terrytangyuan Sep 28, 2021

zw0610 Sep 29, 2021

terrytangyuan Sep 29, 2021

terrytangyuan commented Sep 28, 2021

Jeffwan commented Sep 29, 2021

zw0610 commented Dec 1, 2021

terrytangyuan left a comment

Jeffwan Dec 12, 2021

zw0610 Dec 12, 2021

Jeffwan Dec 12, 2021

zw0610 Dec 12, 2021

Jeffwan Dec 12, 2021

Jeffwan Dec 12, 2021

zw0610 Dec 12, 2021 •

edited

Loading

Jeffwan Dec 12, 2021

Jeffwan commented Dec 12, 2021

Jeffwan commented Dec 12, 2021

zw0610 commented Dec 19, 2021

terrytangyuan commented Dec 19, 2021

terrytangyuan commented Dec 19, 2021

google-oss-prow bot commented Dec 19, 2021

simplify reconciler #166

simplify reconciler #166

Conversation

zw0610 commented Sep 28, 2021 • edited Loading

google-oss-robot commented Sep 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terrytangyuan commented Sep 28, 2021

Jeffwan commented Sep 29, 2021

zw0610 commented Dec 1, 2021

terrytangyuan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zw0610 Dec 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jeffwan commented Dec 12, 2021

Jeffwan commented Dec 12, 2021

zw0610 commented Dec 19, 2021

terrytangyuan commented Dec 19, 2021

terrytangyuan commented Dec 19, 2021

google-oss-prow bot commented Dec 19, 2021

zw0610 commented Sep 28, 2021 •

edited

Loading

zw0610 Dec 12, 2021 •

edited

Loading