feat: add design docs for dataflow affinity using any preceding data operations #68
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…operations Signed-off-by: xliuqq <[email protected]>
cheyang
reviewed
Sep 6, 2024
| name: step1-download-model | ||
| namespace: default | ||
| ... | ||
| --- |
There was a problem hiding this comment.
可以提供一个affinity依赖上一个dataProcess的例子。
cheyang
reviewed
Sep 6, 2024
| ## 设计 | ||
|
|
||
| > 字段改动，因此当前设计对 v1.0.1 和 v1.0.2 的 DataFlow Affinity 不兼容。 | ||
| > |
There was a problem hiding this comment.
是否可以定义一个通用的结构体 ObjectRef 来包含引用操作的必要字段, 这样来保持向前兼容：
type ObjectRef struct {
// API version of the referent operation
// +optional
APIVersion string `json:"apiVersion,omitempty"`
// Kind specifies the type of the referent operation
// +required
// +kubebuilder:validation:Enum=DataLoad;DataBackup;DataMigrate;DataProcess
Kind string `json:"kind"`
// Name specifies the name of the referent operation
// +required
Name string `json:"name"`
// Namespace specifies the namespace of the referent operation.
// +optional
Namespace string `json:"namespace,omitempty"`
}在其他结构体中使用 ObjectRef
然后，在需要引用操作的地方使用这个通用结构体。例如，在 OperationRef 和 AffinityStrategy 中使用 ObjectRef：
type OperationRef struct {
ObjectRef `json:",inline"`
// AffinityStrategy specifies the pod affinity strategy with the referent operation.
// +optional
AffinityStrategy AffinityStrategy `json:"affinityStrategy,omitempty"`
}
type AffinityStrategy struct {
// Specifies the dependent preceding operation in a workflow. If not set, use `RunAfter` field.
// +optional
DependOn *ObjectRef `json:"dependOn,omitempty"`
// Policy one of: "", "Require", "Prefer"
// +optional
Policy AffinityPolicy `json:"policy,omitempty"`
Prefers []Prefer `json:"prefers,omitempty"`
Requires []Require `json:"requires,omitempty"`
}There was a problem hiding this comment.
我的疑问是：
是否保留 AffinityStrategy 定义在 RunAfter: OperationRef 里面？如果是的话，就可以兼容，抽象出 ObjectRef。
@cheyang
There was a problem hiding this comment.
我倾向于AffinityStrategy放在OperationRef中，因为不想提供单独AffinityStrategy，也有含义，也是有runAfter的语义。FYI @TrafalgarZZZ
There was a problem hiding this comment.
是否可以定义一个通用的结构体 ObjectRef 来包含引用操作的必要字段, 这样来保持向前兼容：
type ObjectRef struct { // API version of the referent operation // +optional APIVersion string `json:"apiVersion,omitempty"` // Kind specifies the type of the referent operation // +required // +kubebuilder:validation:Enum=DataLoad;DataBackup;DataMigrate;DataProcess Kind string `json:"kind"` // Name specifies the name of the referent operation // +required Name string `json:"name"` // Namespace specifies the namespace of the referent operation. // +optional Namespace string `json:"namespace,omitempty"` }在其他结构体中使用 ObjectRef 然后，在需要引用操作的地方使用这个通用结构体。例如，在 OperationRef 和 AffinityStrategy 中使用 ObjectRef：
type OperationRef struct { ObjectRef `json:",inline"` // AffinityStrategy specifies the pod affinity strategy with the referent operation. // +optional AffinityStrategy AffinityStrategy `json:"affinityStrategy,omitempty"` } type AffinityStrategy struct { // Specifies the dependent preceding operation in a workflow. If not set, use `RunAfter` field. // +optional DependOn *ObjectRef `json:"dependOn,omitempty"` // Policy one of: "", "Require", "Prefer" // +optional Policy AffinityPolicy `json:"policy,omitempty"` Prefers []Prefer `json:"prefers,omitempty"` Requires []Require `json:"requires,omitempty"` }
这个对于inline字段对于 java/python SDK 是否需要修改？SDK 能兼容吗？
cheyang
reviewed
Sep 9, 2024
| metadata: | ||
| name: step2-trtllm-convert | ||
| # exposed affinity which will be filled in OperationStatus. | ||
| fluid.io/affinity.labels: "node.kubernetes.io/instance-type" |
There was a problem hiding this comment.
data-operation.fluid.io/affinity.labels如何？
Signed-off-by: xliuqq <[email protected]>
|
@cheyang @TrafalgarZZZ 是否还有其它的问题？ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
add design docs for dataflow affinity using any preceding data operations