After the cache disk is recovered, write data times out for 5 minutes

[YunhuiChen]

What happened:

Use writeback mode

1、fio randwrite
fio --name=big-file-multi-write --directory=/var/lib/juicefs/volume/pvc-618dce8f-e2a1-4847-93e8-8a4a7ce247b9-ttbxpf --rw=randwrite --refill_buffers --bs=4k --size=1G --numjobs=1 --end_fsync=1

2、Simulated cache disk failure

• name: juicefs-io-fault
  templateType: IOChaos
  ioChaos:
  action: fault
  mode: one
  selector:
  namespaces:
  - kube-system
  labelSelectors:
  app.kubernetes.io/name: juicefs-mount
  labelSelectors:
  chaostest: "true"
  volumePath: /data/cache
  path: /data/cache/**/*
  errno: 5
  percent: 1

3、fio is stuck. Return to EIO

4、Restore cache disk, wait a few minutes to write fio again, still stuck to return to EIO
log.tar.gz
Uploading log.tar.gz…

5、But after waiting for a long time, I can read and write normally

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
• Cloud provider or hardware configuration running JuiceFS:
• OS (e.g cat /etc/os-release):
• Kernel (e.g. uname -a):
• Object storage (cloud provider and region, or self maintained):
• Metadata engine info (version, cloud provider managed or self maintained):
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[davies]

Should be fixed by #4871