support dataflow affinity using any preceding data operations #4324
Conversation
Signed-off-by: xliuqq <[email protected]>
Signed-off-by: xliuqq <[email protected]>
|
Skipping CI for Draft Pull Request. |
Signed-off-by: xliuqq <[email protected]>
Signed-off-by: xliuqq <[email protected]>
Signed-off-by: xliuqq <[email protected]>
|
| dependOnOp := runAfter.AffinityStrategy.DependOn | ||
| if dependOnOp == nil { | ||
| dependOnOp = &runAfter.ObjectRef | ||
| } | ||
|
|
There was a problem hiding this comment.
Current logic allows users to dependOn a data operation which is not in the dataflow. For example:
apiVersion: data.fluid.io/v1alpha1
kind: DataProcess
metadata:
name: step4-infer-server
spec:
runAfter:
kind: DataLoad
name: step3-warmup-cache
namespace: default
affinityStrategy:
# get affinity from which data operation
dependOn:
kind: DataProcess
name: some-other-op
namespace: default
policy: Require
# Require to run on a node with the same label value as the dependent operation
requires:
- name: node.kubernetes.io/instance-typeIs it by design? If not, is it possible to check if the spec.affinityStrategy.dependOn object is a preceding data operation?
There was a problem hiding this comment.
In the context of location affinity, I think there is indeed an implicit temporal dependency. By specifying affinity, we need ensure that a particular data operation runs on specific nodes where previous operations might have already done.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cheyang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Ⅰ. Describe what this PR does
support dataflow affinity using any preceding data operations, more details see fluid-cloudnative/community#68
Ⅱ. Does this pull request fix one issue?
fixes #XXXX
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
unit test.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews