juicefs 1.3.0

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

Details

release notes

This is the fourth major release of JuiceFS, and is an LTS version as well that will be maintained for 24 months.
Since v1.2.0 it has 696 commits from 60 contributors, thanks to
@zhijian-pro @jiefenghuang @polyrabbit @zhoucheng361 @davies @chenjie4255 @YunhuiChen @tangyoupeng @CodingPoeta @CaitinChen @Xuyuchao-juice @SandyXSD @anysql @xiaogaozi @yuhr123 @zxh326 @kyungwan-nam @Suave @timfeirg @zwwhdls @wenjianhn @winglq @cxljs @dsd2077 @ddrid @daimashusheng @SonglinLife @williamstein @mt5225 @lidaohang @JoyLiuc @Looka149 @YinhaoHu @justforlxz @yetone @wpleonardo @taadis @shenxingwuying @prOOrc @myron369 @menwenjun @hawkli-1994 @bitsark @chnliyong @austinhmh @wzhallright @201341 @servusdei2018 @mrdrivingduck @maxjustus @lianneli @davidzhangbj @sjp00556 @jerrycaimin @aucusaga @solracsf @fuyufjh @veinkr @DASWU @achetronic !
Highlights

Python SDK (#5437,#5516,#5573,#5779,#5852,#5847,#5853,#5931,#5998...)
Improved tools and compatibility on Windows (#5376,#5630,#5629,#5643,#5661,#5714,#5724...)
Speedup backup using protobuf (#5344,#5348,#5352,#5356,#5733,#5379,#5383,#5404...)
Support Apache Ranger (#5285,#5430,#6016)

New

fuse: add --async_dio option to enable asynchronous handling of direct I/O (#4943,#4986)
fuse: add --max-fuse-io option to config max write size for request (#5087,#6017)
fuse: add timeout for FUSE requests (#5670,#5681,#5690)
fuse: enable kernel symlink cache (#5766)
fuse: support /dev/fd/N mount point for --fusemount (#5076,#5082,#5976)
cmd/mount: add max-readahead option to set the max readahead buffer (#5225,#5229,#5704)
cmd/mount: add --sort-dir option to sort entries by name within a directory (#5031)
cmd/mount: add readdir-cache option to enable kernel caching readdir entries (#5462)
cmd/mount: add negative-entry-cache option to allow kernel caching negative dir entry lookups (#5788,#6038)
cmd/mount: add cache-large-write option to cache blocks after uploading (#5261)
cmd/mount: add cache-items option to limit number of cache items (#5489,#5495)
cmd/mount: add check-storage option to perform object storage test at startup (#5281)
cmd/mount: add enable-cap and enable-selinux options to enable corresponding xattr requests (#5406,#5962)
cmd/mount: add all-squash option to map all users to a specified one (#5395)
cmd/mount: add umask option to overwrite the one from app (#5756,#5942)
cmd/mount: add disable-transparent-hugepage option to disable transparent huge page (#6147)
cmd/mount: supervisor process pass signal to service process (#5238,#5245,#5246,#5278,#5624)
vfs: support O_TMPFILE flag (#5309,#5433)
meta: support readdir streaming (#5162,#5207,#5223,#5242,#5482,#5504,#5895)
meta: add fast-statfs option to speed up statfs (#6168,#6171)
meta/tikv: support open-tso-follower-proxy and max-tso-batch-wait-interval options (#6166)
meta/sql: support multiple volumes sharing a single db (#5791,#5804)
cache: add env JFS_DROP_OSCACHE to enable dropping page cache (#4981)
cmd/format: support TLS configuration via bucket URL parameters (#5241)
cmd/sync: add max-failure option to abort on error threshold (#5142)
cmd/sync: add start-time and end-time options to filter file (#6072,#6121)
cmd/sync: add check-change option to verify file size (#6093)
cmd/quota: add --create option to create directory if not exists (#5196)
cmd/rmr: add skip-trash option to delete files directly (#5384,#5818)
cmd: add threads option for rmr tool, webdav, gateway and libjfs to delete files (#5522,#5535)
cmd/gateway: add object-meta option to enable object metadata api (#5536)
cmd/sync: add files-from option to specify the list of files or directories to sync (#5639,#5864,#5894)
cmd/objbench: add shards to support object storage shards (#5755)
cmd/dump&load: support zstd compression (#4921)
cmd/webdav: support PROPPATCH method (#5041)
sdk/hadoop: support subdir option in hadoop sdk (#6097,#6141)

Changed

vfs: limit buffer usage for read requests (#5557)
meta: improve statfs performance (#5161)
meta/kv: improve mknod performance via batch get (#5056)
meta: improve performance via inode prefetching. (#5130,#5481)
meta: reduce GetAttr calls (#5540)
meta: optimizations for reducing transaction conflicts (#5308,#5330,#5574)
meta: adjust the interval to one hour for deleted files cleanup (#5343)
meta: cancel background tasks when closing session (#5486,#5613,#5943,#5986,#6108,#6060)
meta/kv: improve scan performance and robustness (#5298,#5314,#5080)
meta/kv: simplify transactions in point get scenarios to reduce one RPC call for TiKV (#6136,#6150)
meta/sql: improve performance by simplifying transaction and reducing locks (#5377,#5460)
meta/sql: set default transaction isolation level to repeatable-read (#5432,#5456)
meta/sql: support more connection options (#5512,#5806)
meta/sql: improve compatibility by updating exact required fields (#6153,#6158)
meta/redis: support tls server name configuration (#5182)
cache: inherit atime and track logic size when re-scan (#5692)
cache: adjust cache capacity and max items by freeratio (#5970)
cache: change the default value of verify-cache-checksum option to extend (#6103)
cmd: exit compact timely and log a message when long-running background tasks start (#6033)
cmd: refuse more than expected positional arguments (#6154)
cmd/mount: the upload-hours option allow using - as a separator (#5494)
cmd/mount: set the process as an IO_FLUSHER to avoid potential hang (#6144,#6157)
cmd/gateway: add one more hierarchy to temporary multipart dir to reduce transaction conflicts (#5549)
cmd/gateway: improve the gateway performance in read-only mode (#6187)
cmd/gateway: some performance optimizations (#5163,#5548,#5544)
cmd/gateway: delete intermediate temporary files promptly to reduce inode usage (#6080)
cmd/gateway: implement ListObjectVersions (#6173)
cmd/sync: support more environment variables in cluster mode (#5364,#5422)
cmd/sync: improve performance by calculating checksum in flight to reduce redundant reads from the source (#5524)
cmd/warmup: accelerate big file warmup (#5713,#5886,#5897)
cmd/warmup: improve performance by reducing file operations (#5408)
cmd/warmup: show cache location info (#5559)
cmd/info: add column for absolute file offset (#6037)
cmd/fsck: support recalculating used space and inodes (#5331,#6036)
cmd/restore: output restored dirs and stats (#5380)
cmd/gc: use MaxDeletes as remove concurrency (#5532)
cmd/gc: improve scanning performance for pending-deletion files (#5683)
cmd/backup: use multipart upload to transfer large backup files (#5564)
cmd/stats: add timestamp info and count option (#5528)
cmd/debug: collect more detailed information about goroutines (#5656)
object: upgrade aws sdk to v2 (#5968,#5987,#5992)
object: remove object storage support for scs, jss, upyun (#5972, #5975, #5974)
object: improve list performance via v2 API (#5250,#5393,#5434)
object: display full address of s3 compatible object storage in log (#6151)
object: disable s3 sdk retries since JuiceFS has retry mechanism (#5644,#6113)
object/gs: improve gs performance by using client pool (#5020)
object/nfs: improve Readdir performance by increasing batch size (#5689)
object/minio: support REDUCED_REDUNDANCY storage class for MinIO (#5413)
object/sftp: supports verification of known hosts (#5581)
object/bos: reduce memory allocation in bos (#5657)
object/bos: support crc32 check (#5883)
object/oss: support parsing regionId from environment variable and from VPC-type endpoint (#6202)
hadoop: support skipNBytes (#5028)
metrics: update grafana dashboard (#4980,#5248)
metrics: add method label for transaction conflicts (#5762)
metrics: add total_inodes and total_space metrics (#5424)
metrics: expand the FUSE latency histogram range (#6199)

Bugfix

vfs: remove object files if metadata commit fails with ENOENT (#5133)
vfs: allow ftruncate on an opened and deleted file (#5664)
meta: fix the issue that trash files may be dangling (#5414)
meta: evict symlink cache periodically (#5131)
meta: fix permission check of open and mknod (#4948)
meta: fix volume stats update issues (#4979,#5221,#5272,#5280,#5303,#5305)
meta: fix the issue that the username cannot contain the "@" character (#5761)
meta: fix potential issues with context reuse (#5372,#5665)
meta: exit doCleanupSlices timely (#6081)
meta: check for incorrect filenames (#6073)
meta: fix stats for trash directories (#6099)
meta: fix all-zero read response due to invalid openFile.chunks cache (#6192)
meta: fix leaking objects when quota is full (#6200)
meta/kv: return EINVAL for empty xattr value when using TiKV (#6104)
meta/sql: fix nanosecond of link and readlink (#5496)
meta/sql: fix sqlite3 "database is locked" issue (#5586)
meta/sql: fix flush quotas and stats deadlock (#5706,#5712)
meta/sql: fix database retry issues (#5440,#5485)
meta/sql: fix concurrency issue in json backup dump (#5930)
meta/redis: fix potential loss of attributes issue (#5709)
meta/redis: check MGet error to avoid panic (#5773)
meta/redis: fix parsing options for redis cluster (#6129)
chunk: remove unexpected read retries (#5199)
chunk: deduplicate prefetch/partial-block/full-block requests to avoid read amplification (#4947，#5219)
cache/stage: retry uploading stage blocks upon disk read failures (#5084)
cache/stage: add stage write timeout with object storage fallback to prevent hangs (#5230,#5563,#5575)
cache: disable writeback during compaction to prevent EIO from multi-client races (#5767)
cache: maintain a margin based on the free-space ratio during cleanup to prevent client interference (#5216)
cache: fix inoperative --cache-expire option in memory cache (#5313)
cache: fix cache duplication issue caused by concurrency (#5623)
cache: fix potential high latency caused by locks in getDiskUsage (#5857)
cache: fix sleep duration in cache expiration loop (#6165)
cmd/mount: fix process management issues (#5064,#5172,#5641,#5881)
cmd/mount: fix potential hangs during stack trace collection by adding timeout (#5325)
cmd/mount: support using ":" to separate multiple cache-dir in fstab (#5719)
cmd/mount: fix the mount path to an absolute path in fstab (#5951)
cmd/debug: fix potential hangs by adding timeout (#5050,#5627)
cmd/sync: fix command-line interaction issues (#5138,#5744,#5752,#6115)
cmd/sync: fix parallel list issues (#5645,#5646)
cmd/sync: fix statistical issues (#5658,#5763)
cmd/sync: fix manager ip issues (#5597,#5950,#5947)
cmd/sync: ignore special files of JuiceFS (#5523)
cmd/sync: fix leaked directories issue (#5705,#5777,#5787)
cmd/sync: fix missing error log for worker process (#5362)
cmd/sync: use a dedicated buffer pool to avoid contention with writes (#5568)
cmd/sync: fix the issue that worker might get an outdated binary. (#5899)
cmd/sync: fix check-all in cluster mode (#5904)
cmd/sync: fix uneven task distribution among workers. (#5957)
cmd/sync: return error if uid or gid cannot be found (#5995)
cmd/sync: fix copied counter (#6114)
cmd/sync: fix mtime comparison for check-change (#6134)
cmd/sync: fix progress bar for file list (#6162)
cmd/gateway: fix missing memory buffer usage metrics (#5560)
cmd/gateway: fix potential etag inconsistency caused by concurrency (#5258,#5378)
cmd/gateway: fix some compatibility issues (#6004,#6006)
cmd/gateway: fix some panic issues (#5098, #5164,#5807)
cmd/fsck: fix the missing trash files statistics in volume stat (#6105)
cmd/clone: skip trash files (#5109)
cmd/gc: fix misclassification of pending-delete objects as leaked (#5198)
cmd/config: fix bucket update issue (#5653)
cmd/objbench: fix time and size statistics (#5264,#5267,#6201)
cmd/warmup: fix handling files with holes (#6034)
cmd/warmup: prevent division by zero when checking for empty files or directories (#6074)
object: fix error handling for directory creation and deletion (#6176)
object: fix burst bandwidth (#6102)
object: fix the issue where modifying mtime also changes atime (#6026)
object: fix sha256 checksum in s3 is added again (#6100)
object: add validation for ciphertext length to prevent panic caused by data corruption in object storage (#6107)
object: fix error handling in HEAD (#6120)
object: fix object mtime precision issue (#6127)
object/sftp: fix connection reuse issue (#6123)
object/ks3: check endpoint format to avoid panic (#5074)
object/s3: fix long timeout issue when AWS IMDSv2 is disabled (#5231)
object/tos: fix wrong region of TOS (#6191)
object/jfs: fix file creation logic (#5792)
hadoop: fix juicefs.umask does not work (#5337)
hadoop: fix background task thread leak (#5363)
hadoop: fix HDFS command not displaying JuiceFS quota (#5939)
utils/cond: fix timer issue (#5631,#5634)
log: fix quote parsing issue in accesslog (#5588)
build: fix static linking for ARM64 binary (#5612)

Deps

deps: bump github.com/winfsp/cgofuse from v1.5.0 to v1.6.0 (#5840)
deps: bump MySQL driver github.com/go-sql-driver/mysql from v1.7.1(2023-04-25) to v1.9.1(2025-03-20) (#5834)
deps: bump PostgreSQL driver github.com/jackc/pgx/v5 from v5.5.4 to v5.7.3 (#5835)
deps: bump sqlite3 driver github.com/mattn/go-sqlite3 from v1.14.16 to v1.14.24 (#5555)
deps: bump BadgerDB github.com/dgraph-io/badger/v4 from v4.2.0 to v4.5.1 (#5626)
deps: bump Redis client github.com/redis/go-redis/v9 from v9.0.2 to v9.7.3 (#5832)
deps: bump TiKV SDK from v2.0.1 to v2.0.7 (#5850)
deps: bump baidu bce sdk from v0.9.200 to v0.9.221 (#5820)
deps: bump tencent cos sdk to v0.7.45 to v0.7.63 (#5821)
deps: bump volcengine tos sdk from v2.7.0 to v2.7.8 (#5838)
deps: change aliyun oss sdk to github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.2.1 (#5830,#5878)
deps: bump ks3 sdk from v1.2.2 to v1.6.0 (#5829)
deps: bump ibm cos sdk from v1.10.0 to v1.12.1 (#5828)
deps: bump qiniu sdk from v7.15.0 to v7.25.2 (#5831)
deps: bump gs sdk from v1.28.1(2022-12-02) to v1.48.0(2024-12-05) (#5849)
deps: bump azure sdk from v1.0.0 to 1.3.1 (#5819)
deps: bump github.com/ncw/swift/v2 from v2.0.1 to v2.0.3 (#5836)
deps: bump github.com/colinmarc/hdfs/v2 from v2.3.0 to v2.4.0 (#5827)
deps: bump github.com/studio-b12/gowebdav from 3282f94193f2(2023-02-04) to v0.10.0 (#5839)
deps: bump prometheus from v1.15.1 to v1.21.1 (#5843)
deps: bump github.com/goccy/go-json from v0.10.2 to v0.10.5 (#5841)
deps: bump github.com/grafana/pyroscope-go from v1.0.4 to v1.2.1 (#5842)
deps: bump golang.org/x/crypto from 0.21.0 to 0.31.0 (#5374)
deps: bump golang.org/x/net from v0.23.0 to v0.33.0 (#5412)
deps: bump github.com/DataDog/zstd from v1.5.0 to v1.5.6 (#5741)
deps: bump github.com/remyoudompheng/bigfft to latest version (#5747)
deps: bump go.uber.org/automaxprocs from v1.5.2 to v1.6.0 (#5844)

View the full release notes at https://github.com/juicedata/juicefs/releases/tag/v1.3.0.

---

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.