Invalid meta driver: $'postgres [interface.go:500]

[KKulishov]

What happened:

I deploy juicefs csi on k8s and use S3 (Minio) as object storage and cockroachdb as metadata engine.
And I create pvc and pod that pvc created successfully but pod cannnot attach volume.

describe pod

MountVolume.SetUp failed for volume "juicefs-pv" : rpc error: code = Internal desc = Could not mount juicefs: juicefs format error: 2024/06/27 08:14:29.156229 juicefs[97] <INFO>: Meta address: $'postgres://jfs:****@urt-jfs-db1-lb1.xxxxxx.group:26257/jfs' [interface.go:497] 2024/06/27 08:14:29.156485 juicefs[97] <FATAL>: Invalid meta driver: $'postgres [interface.go:500] : exit status 1

logs juicefs-csi-node

I0627 05:15:17.460443       7 node.go:106] NodePublishVolume: volume_id is juicefs-pv
I0627 05:15:17.460475       7 node.go:117] NodePublishVolume: volume_capability is mount:<fs_type:"juicefs" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > 
I0627 05:15:17.460528       7 node.go:123] NodePublishVolume: creating dir /var/lib/kubelet/pods/912e9acc-a963-4d08-b65e-5735641b9a50/volumes/kubernetes.io~csi/juicefs-pv/mount
I0627 05:15:17.460735       7 node.go:138] NodePublishVolume: volume context: map[csi.storage.k8s.io/ephemeral:false csi.storage.k8s.io/pod.name:app-test-5cd58d95bc-vsj7n csi.storage.k8s.io/pod.namespace:infra csi.storage.k8s.io/pod.uid:912e9acc-a963-4d08-b65e-5735641b9a50 csi.storage.k8s.io/serviceAccount.name:default]
I0627 05:15:17.460768       7 node.go:148] NodePublishVolume: mounting juicefs with secret [metaurl name secret-key storage access-key bucket envs format-options], options []
I0627 05:15:17.461103       7 juicefs.go:1036] ceFormat cmd: [/usr/local/bin/juicefs format --storage=s3 --bucket=https://k8s-test-new.s3-jfs.xxxx.group --access-key=k8s-test-new --secret-key=${secretkey} ${metaurl} myjfs --trash-days=1 --block-size=4096]
I0627 05:15:17.575567       7 juicefs.go:1056] Format output is 2024/06/27 08:15:17.561114 juicefs[125] <INFO>: Meta address: $'postgres://jfs:****@urt-jfs-db1-lb1.xxxx.group:26257/jfs' [interface.go:497]
2024/06/27 08:15:17.561475 juicefs[125] <FATAL>: Invalid meta driver: $'postgres [interface.go:500]
I0627 05:15:17.575621       7 juicefs.go:1059] Format error: exit status 1
E0627 05:15:17.575682       7 driver.go:102] GRPC error: rpc error: code = Internal desc = Could not mount juicefs: juicefs format error: 2024/06/27 08:15:17.561114 juicefs[125] <INFO>: Meta address: $'postgres://jfs:****@urt-jfs-db1-lb1.xxxxx.group:26257/jfs' [interface.go:497]
2024/06/27 08:15:17.561475 juicefs[125] <FATAL>: Invalid meta driver: $'postgres [interface.go:500]
: exit status 1

Linux version in juicefs-csi-node : Oracle Linux Server 8.7 and kernel: 5.4.17-2136.315.5.el8uek.x86_64

Install JuiceFS helm (this repo) and set in values.yaml
mountMode: process

and changes

node:
  enabled: true
  debug: false
  hostNetwork: true

this my StrorageClass

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: jucefs
    meta.helm.sh/release-namespace: infra
  creationTimestamp: "2024-06-27T05:50:47Z"
  labels:
    app.kubernetes.io/instance: jucefs
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: 0.24.1
    helm.sh/chart: juicefs-csi-driver-0.20.3
  name: juicefs-sc
  resourceVersion: "2672155149"
  uid: f149a442-3db1-416d-9450-a7728a9f8def
mountOptions:
- writeback
- upload-delay=1m
parameters:
  csi.storage.k8s.io/controller-expand-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: infra
  csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/node-publish-secret-namespace: infra
  csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/provisioner-secret-namespace: infra
  juicefs/mount-annotations: '{"prometheus.io/port":"9567","prometheus.io/scrape":"true"}'
  juicefs/mount-cpu-limit: 4000m
  juicefs/mount-cpu-request: 100m
  juicefs/mount-memory-limit: 5Gi
  juicefs/mount-memory-request: 100Mi
  pathPattern: ${.PVC.annotations.csi.juicefs.com/subdir}
provisioner: csi.juicefs.com
reclaimPolicy: Retain
volumeBindingMode: Immediate

this my secrets

apiVersion: v1
kind: Secret
metadata:
  name: juicefs-sc-secret
  namespace: infra
type: Opaque
stringData:
  name: myjfs              
  access-key: k8s-test-new       
  secret-key: xxxxxxxxxxxxxxxxxxxxxxxxxxx       
  metaurl: postgres://jfs:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@urt-jfs-db1-lb1.xxxxxxxxxxxx.group:26257/jfs          
  storage: s3                   
  bucket: https://k8s-test-new.s3-jfs.xxxxxxxx.group               
  format-options: trash-days=1,block-size=4096

this my test app:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: juicefs-pv
  labels:
    juicefs-name: ten-pb-fs
spec:
  # For now, JuiceFS CSI Driver doesn't support setting storage capacity for static PV. Fill in any valid string is fine.
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    # A CSIDriver named csi.juicefs.com is created during installation
    driver: csi.juicefs.com
    # volumeHandle needs to be unique within the cluster, simply using the PV name is recommended
    volumeHandle: juicefs-pv
    fsType: juicefs
    # Reference the volume credentials (Secret) created in previous step
    # If you need to use different credentials, or even use different JuiceFS volumes, you'll need to create different volume credentials
    nodePublishSecretRef:
      name: juicefs-sc-secret
      namespace: infra
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  namespace: infra
  annotations:
    csi.juicefs.com/additional-mount-options: "writeback,upload-delay=1m" # Add JuiceFS settings if necessary.
    juicefs/mount-cpu-limit: 4000m
    juicefs/mount-cpu-request: 100m
    juicefs/mount-memory-limit: 5Gi
    juicefs/mount-memory-request: 100Mi
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  # Must use an empty string as storageClassName
  # Meaning that this PV will not use any StorageClass, instead will use the PV specified by selector
  storageClassName: ""
  # For now, JuiceFS CSI Driver doesn't support setting storage capacity for static PV. Fill in any valid string that's lower than the PV capacity.
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      juicefs-name: ten-pb-fs
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-test
  name: app-test
  namespace: infra
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-test
  template:
    metadata:
      labels:
        app: app-test
      annotations:
        app: app-test
    spec:
      containers:
        - args:
          - -c
          - while true; do echo $(date -u) >> /data/out.txt; sleep 5; done
          command:
          - /bin/sh
          image: busybox
          name: app
          volumeMounts:
          - mountPath: /data
            name: data
          resources:
            limits:
              cpu: 400m
              memory: 1Gi
            requests:
              cpu: 100m
              memory: 100Mi
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: juicefs-pvc

But if I change in values.yaml from process on mountpod

mountMode: mountpod

then everything is ok. PV and PVC are being created and mounted pods

can you help solve this case? Why mode "mountMode: process" don't mount.

[zhijian-pro]

@KKulishov We recently fixed a bug juicedata/juicefs#4731 with postgres not sure if it's related.
You can download the juicefs v1.2 binary and try it with the same metaurl format.
The postgresql format doc: https://juicefs.com/docs/community/databases_for_metadata/#postgresql

[KKulishov]

@zhijian-pro thanks for the advice

But version

I use this helm

image

image:
  repository: juicedata/juicefs-csi-driver
  tag: "v0.24.1"
  pullPolicy: ""

dashboardImage:
  repository: juicedata/csi-dashboard
  tag: "v0.24.1"
  pullPolicy: ""

sidecars:
  livenessProbeImage:
    repository: registry.k8s.io/sig-storage/livenessprobe
    tag: "v2.12.0"
    pullPolicy: ""
  nodeDriverRegistrarImage:
    repository: registry.k8s.io/sig-storage/csi-node-driver-registrar
    tag: "v2.9.0"
    pullPolicy: ""
  csiProvisionerImage:
    repository: registry.k8s.io/sig-storage/csi-provisioner
    tag: "v2.2.2"
    pullPolicy: ""
  csiResizerImage:
    repository: registry.k8s.io/sig-storage/csi-resizer
    tag: "v1.9.0"
    pullPolicy: ""

Can you suggest which image should be used for

mountMode: process

I repeat , if I set the mode

mountMode: mountpod

everything is working . PV and PVS are created and mounted in pods.

When user (mountMode: mountpod)

image use pods juicefs-urt-k8s-test-w8-pvc-b4a40f88-a9e2-4573-9233-b85434fc8a6d-wwjuaf

juicedata/mount:ce-v1.1.2

version jucefsversion

The problem only occurs when I use

mountMode: process

there may be a problem in the operating logic in https://github.com/juicedata/charts/blob/main/charts/juicefs-csi-driver/templates/daemonset.yaml

[zhijian-pro]

Can you desensitize the meta-url and post it, I need to determine if his formatting is correct.

[KKulishov]

@zhijian-pro Sorry, I can't post url. It my corp network. I can share logs only

but I changed cocroachdb on redis

use

mountMode: process

and with redis everything worked

rather the problem is on the side controller

[polyrabbit]

@zhijian-pro @zxh326 I suppose it's been escaped by this function?

[zxh326]

Does your pg password contain the '$', '&', ';', '>', '|', '(', ')'?

Special characters in the password need to be replaced by url encoding. For example, | needs to be replaced with %7C

[KKulishov]

Thank you @zxh326 @polyrabbit . the problem was with the passwords. the line contained a special character '&' , changed to '%26' and redeploy.