Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix multi cache dir #2639

Merged
merged 8 commits into from
Feb 27, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions docs/zh/samples/juicefs/juicefs_cache_dir.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# JuiceFSRuntime 缓存配置

如何在 Fluid 中使用 JuiceFS,请参考文档[示例 - 如何在 Fluid 中使用 JuiceFS](juicefs_runtime.md)。本文讲述所有在 Fluid 中有关 JuiceFS 的缓存相关配置。

## 设置多个路径缓存

缓存路径在 JuiceFSRuntime 中的 tiredstore 设置,worker 和 fuse pod 共享相同的配置。

注意:JuiceFS 支持多路径缓存,不支持多级缓存。

```yaml
apiVersion: data.fluid.io/v1alpha1
kind: JuiceFSRuntime
metadata:
name: jfsdemo
spec:
replicas: 1
tieredstore:
levels:
- mediumtype: SSD
path: /mnt/cache1:/mnt/cache2
quota: 40Gi
low: "0.1"
```

其中:
- `spec.tiredstore.levels.path` 可设置为多个路径,以 `:` 分隔,缓存会被分配在这里设置的所有路径下;但不支持通配符;
- `spec.tiredstore.levels.quota` 为缓存对象的总大小,与路径多少无关;
- `spec.tiredstore.levels.low` 为缓存路径的最小剩余空间比例,无论缓存是否达到限额,都会保证缓存路径的剩余空间;
- `spec.tiredstore.levels.mediumtype` 为缓存路径的类型,目前支持 `SSD` 和 `MEM`。


## 单独设置 worker 的缓存路径

默认情况下,worker 和 fuse 的缓存路径都在 `spec.tiredstore.levels.path` 中设置,但是也可以单独设置 worker 的缓存路径。

```yaml
apiVersion: data.fluid.io/v1alpha1
kind: JuiceFSRuntime
metadata:
name: jfsdemo
spec:
worker:
options:
"cache-dir": "/mnt/cache1:/mnt/cache2"
tieredstore:
levels:
- mediumtype: MEM
path: /dev/shm
quota: 500Mi
low: "0.1"
```

其中:
- `spec.worker.options` 为 worker 的挂载参数,缓存路径以 `cache-dir` 为 key,以 `:` 分隔的多个路径;


Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Redis、MySQL、TiKV 等多种数据库中。

## 部署 JuiceFSRuntime 环境

具体部署方法参考文档 [如何在 Fluid 中使用 JuiceFS](./juicefs_runtime.md)。
具体部署方法参考文档 [如何在 Fluid 中使用 JuiceFS](juicefs_runtime.md)。

在 JuiceFSRuntime 和 Dataset 创建成功后,等待 worker pod 启动成功,再进行下面的步骤。

Expand Down Expand Up @@ -63,4 +63,4 @@ root@jfsdemo-worker-0:~#

可以看到 bucket 中的文件已经被同步到了 JuiceFS 中。

最后创建业务 Pod,其中 Pod 使用上面创建的 `Dataset` 的方式为指定同名的 PVC。该步骤与文档 [如何在 Fluid 中使用 JuiceFS](./juicefs_runtime.md) 中一致,这里不再赘述。
最后创建业务 Pod,其中 Pod 使用上面创建的 `Dataset` 的方式为指定同名的 PVC。该步骤与文档 [如何在 Fluid 中使用 JuiceFS](juicefs_runtime.md) 中一致,这里不再赘述。
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ JuiceFS 是一款面向云环境设计的开源高性能共享文件系统,提

## 安装

您可以从 [Fluid Releases](https://github.com/fluid-cloudnative/fluid/releases) 下载最新的 Fluid 安装包。参考 [安装文档](../userguide/install.md) 完成安装。并检查 Fluid 各组件正常运行:
您可以从 [Fluid Releases](https://github.com/fluid-cloudnative/fluid/releases) 下载最新的 Fluid 安装包。参考 [安装文档](../../userguide/install.md) 完成安装。并检查 Fluid 各组件正常运行:

```shell
$ kubectl get po -n fluid-system
Expand Down
16 changes: 8 additions & 8 deletions pkg/controllers/v1alpha1/databackup/implement.go
Original file line number Diff line number Diff line change
Expand Up @@ -433,14 +433,14 @@ func (r *DataBackupReconcilerImplement) generateDataBackupValueFile(ctx reconcil
imagePullSecrets := docker.GetImagePullSecretsFromEnv(common.EnvImagePullSecretsKey)

dataBackup := cdatabackup.DataBackup{
Namespace: databackup.Namespace,
Dataset: databackup.Spec.Dataset,
Name: databackup.Name,
NodeName: nodeName,
Image: image,
JavaEnv: javaEnv,
Workdir: workdir,
RuntimeType: runtimeType,
Namespace: databackup.Namespace,
Dataset: databackup.Spec.Dataset,
Name: databackup.Name,
NodeName: nodeName,
Image: image,
JavaEnv: javaEnv,
Workdir: workdir,
RuntimeType: runtimeType,
ImagePullSecrets: imagePullSecrets,
}
pvcName, path, err := utils.ParseBackupRestorePath(databackup.Spec.BackupPath)
Expand Down
8 changes: 6 additions & 2 deletions pkg/ddc/juicefs/transform_fuse.go
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,11 @@ func (j *JuiceFSEngine) genValue(mount datav1alpha1.Mount, tiredStoreLevel *data
var storagePath = DefaultCacheDir
var volumeType = common.VolumeTypeHostPath
if tiredStoreLevel != nil {
storagePath = tiredStoreLevel.Path // /mnt/disk1/bigboot or /mnt/disk1/bigboot,/mnt/disk2/bigboot
// juicefs cache-dir use colon (:) to separate multiple paths
// community doc: https://juicefs.com/docs/community/command_reference/#juicefs-mount
// enterprise doc: https://juicefs.com/docs/cloud/commands_reference#mount
// /mnt/disk1/bigboot or /mnt/disk1/bigboot:/mnt/disk2/bigboot
storagePath = tiredStoreLevel.Path
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix unit test. thanks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if tiredStoreLevel.Quota != nil {
q := tiredStoreLevel.Quota
// juicefs cache-size should be integer in MiB
Expand All @@ -244,7 +248,7 @@ func (j *JuiceFSEngine) genValue(mount datav1alpha1.Mount, tiredStoreLevel *data
}
volumeType = tiredStoreLevel.VolumeType
}
originPath := strings.Split(storagePath, ",")
originPath := strings.Split(storagePath, ":")
options["cache-dir"] = storagePath

// transform cacheDir
Expand Down
2 changes: 1 addition & 1 deletion pkg/ddc/juicefs/transform_volume.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ func (j *JuiceFSEngine) transformWorkerCacheVolumes(runtime *datav1alpha1.JuiceF
}
}
if cacheDir != "" {
originPath := strings.Split(cacheDir, ",")
originPath := strings.Split(cacheDir, ":")
for i, p := range originPath {
var volumeType = common.VolumeTypeHostPath
caches[strconv.Itoa(i+1)] = cache{
Expand Down
4 changes: 2 additions & 2 deletions pkg/ddc/juicefs/transform_volume_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ func TestJuiceFSEngine_transformWorkerCacheVolumes(t *testing.T) {
runtime: &datav1alpha1.JuiceFSRuntime{
Spec: datav1alpha1.JuiceFSRuntimeSpec{
Worker: datav1alpha1.JuiceFSCompTemplateSpec{
Options: map[string]string{"cache-dir": "/worker-cache1,/worker-cache2"},
Options: map[string]string{"cache-dir": "/worker-cache1:/worker-cache2"},
},
},
},
Expand Down Expand Up @@ -336,7 +336,7 @@ func TestJuiceFSEngine_transformWorkerCacheVolumes(t *testing.T) {
runtime: &datav1alpha1.JuiceFSRuntime{
Spec: datav1alpha1.JuiceFSRuntimeSpec{
Worker: datav1alpha1.JuiceFSCompTemplateSpec{
Options: map[string]string{"cache-dir": "/worker-cache1,/worker-cache2"},
Options: map[string]string{"cache-dir": "/worker-cache1:/worker-cache2"},
VolumeMounts: []corev1.VolumeMount{
{
Name: "cache",
Expand Down
3 changes: 2 additions & 1 deletion test/prow/juicefs_access_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ def create_dataset_and_runtime(dataset_name):
"metadata": {"name": dataset_name, "namespace": APP_NAMESPACE},
"spec": {
"replicas": 1,
"tieredstore": {"levels": [{"mediumtype": "MEM", "path": "/dev/shm", "quota": "40960", "low": "0.1"}]}
"tieredstore": {"levels": [
{"mediumtype": "MEM", "path": "/dev/shm/cache1:/dev/shm/cache2", "quota": "40960", "low": "0.1"}]}
}
}

Expand Down