v0.17.5

JuiceFS v0.17.5 is a patch release for v0.17, which has the following changes:

• Sync command crashes for broken symlinks (#1028).
• Fixed the returned code when open a non-existed file (#1035).
• Fixed the leaked key after cleanup stale sessions in TiKV (#1041).
• Fixed sync from/to some prefix of MinIO (#1055).
• Fixed the problem that extended attributes of hard links are removed unexpectedly (#1078).
• Fixed the wrong used space in df (introduced in v0.17.2) (#1096).
• Fixed a bug in gc command that could corrupt file in Redis (#1110).

Thanks to @davies @SandyXSD @chiyutianyi for contributions!

v0.17.2

Changelog

JuiceFS v0.17.2 is the second patch release for v0.17, which has the following changes:

• Fixed potential data leak for SQL engine (#931 #1018).
• Fixed a bug in CopyFileRange() (77a3a62).
• Fixed authentication for GCS in container (#933).
• Don't fill all the disk space in writeback mode(#943).
• Fixed parsing address for Redis Sentinel (#953).
• Added a workaround to sync with object storages that are not compatible with S3 (#984).
• Increase backoff for Redis/TiKV to avoid potential failure under high contention (#999).
• Added a check for AK/SK for S3 gateway (#1001).
• Added an option to provide a customized endpoint for object storage (#1008).
• Chnaged frsize to 4096 to be compatible with local filesystem (#1016).

Thanks to @SandyXSD @davies @chiyutianyi @zhijian-pro @nsampre for contributions!

v0.17.1

JuiceFS v0.17.1 is a patch release for v0.17, which has the following changes:

• Return ENOSYS if attr is disabled (#863).
• Ignore unknown flags in setattr() (#862).
• Sort files when list a directory in Java SDK to be compatible with HDFS (#889).
• Upgrade nats-server to v2.2+ to address CVE-2021-3127 (#893).
• Enable metrics for S3 gateway (#897).
• Fixed WebDAV backend (#899).
• Refresh cache once new attributes found (#902).
• Upgrade dgrijalva/jwt-go to golang-jwt/jwt to address CVE-2020-26160 (#903)
• Fixed parent of renamed files/dirs (#905).
• Fixed chunk deletions for SQL engine (#904)
• Upgrade gjson to fix CVE-2021-42836 (#912).

Thanks to contributions from @SandyXSD @tangyoupeng @davies @zhijian-pro @chiyutianyi !

v0.17.0

JuiceFS v0.17 arrived one month after 0.16, with 80+ commits from 9 contributors (@SandyXSD, @davies, @xiaogaozi, @yuhr123, @Suave @tangyoupeng @201341 @zwwhdls @allwefantasy), thanks to them!

This release improved the performance when JuiceFS is used for temporary data by using an in-memory meta engine (memkv) and delayed uploading. For example, we can use JuiceFS as the shuffle and spill disk without worrying about running out of space.

Linux Test Project was used to verify the compatibility of JuiceFS, please checkout the current results here.

This release introduced metadata cache for Java SDK and S3 gateway (similar to metadata cache in kernel), which could be turned to improve performance significantly.

New

• Added a option to delay upload in writeback mode (#736 #743), which is useful for temporary data.
• Added an in-memory meta engine (memkv://) for temporary data (#751 #779).
• Support CREATE and REPLACE flag in setxattr (#770).
• Added metrics for in-memory cache (#776).
• Support rename flags (NoReplace and Exchange) (#787).
• New colorful result for bench command (#810).
• Added entry and attributes cache for Java SDK and S3 gateway (#835).

Changed

• Default logging directory in macOS was changed to user's home directory (#744).
• Limit the number of retry for listing to 3 to avoid infinite loop (#745).
• Show total size of valid objects in gc command (#746).
• Disable SHA256 checksum for S3 and other compatible object store to reduce CPU usage (#754).
• Hide operations on internal files from access log (#766).
• Require Go 1.15 to build JuiceFS and build the release with Go 1.16 (#771).
• Inherit gid of parent when SGID is set (#772).
• Keep SGID when file is non-group-executable (#773).
• Allow remove broken dir/file (#784).
• Retry transactions for TxnLockNotFound from TiKV (#789).
• Cleanup current session during umount (#796).
• Reduce memory allocation for OBS and Webdav backend (#800).
• Support escaped access key for KS3 (#830).
• Support lookup . and .. (#842).
• No warning if compaction fail with missing objects(#844).
• Increase available inodes based on current usage (#851).
• Allow update access key and secret key with default compress algorithm (#855).

Bugfix

• Fixed a leak in SQL engine (#728).
• Fixed a bug that may crash client (#729).
• Fixed valid bytes of progress bar for gc command (#746).
• Fixed warmup with a long list of files (#752).
• Fixed supporting of secured Redis connection (regression in v0.16) (#758).
• Fixed data corruption in SQL and TiKV engine when some slices are skipped during compaction (#759).
• Fixed metrics for read bytes in Hadoop SDK (#761).
• Fixed multipart upload in S3 gateway (#765).
• Fixed POSIX locks on interweaved regions (#769).
• Fixed latency metrics for libRADOS (#793).
• Fixed concat in Java SDK and multipart upload (#817).
• Fixed nlink of parent when rename directories (#839).
• Fixed transaction for read-only mount (#844).

v0.16.2

JuiceFS v0.16.2 is patched version for v0.16, is recommended to upgrade.

Bugfix

• Retries LIST three times to avoid infinite loop (#745).
• Fixed valid bytes of progress bar for gc command (#746).
• Fixed warmup with a long list of files (#752).
• Fixed supporting of secured Redis connection (regression in v0.16) (#758).
• Fixed data corruption in SQL and TiKV engine when some slices are skipped during compaction (#759).
• Fixed metrics for read bytes in Hadoop SDK (#761).

v0.16.1

JuiceFS v0.16.1 arrived one month after 0.15.2, with 80+ commits from 11 contributors (@davies, @Sandy4999 , @xiaogaozi @tangyoupeng @zhijian-pro @chnliyong @Suave @themaxdavitt @xuhui-lu @201341 @zwwhdls ), thanks to them!

The biggest feature is supporting TiKV as the meta engine, which is a distributed transactional key-value database. With TiKV, JuiceFS can store trillions of files and exabytes of data, please .

BREAKING CHANGE

The meaning of password in Redis Sentinel URL is changed from Sentinel password to Redis server password, please update the password in the URL if you use sentinel and their password are different.

New

• Supports TiKV as meta engine and object store (#610 #629 #631 #636 #645 #633 #663 #666 #675 #704).
• Added limit for upload/download bandwidth (#611).
• Added a virtual file (.config) to show current configurations (#652).
• Added a subcommand stats to watch performance metrics in realtime (#702 #721).
• Added progress bar for fsck, gc, load and dump (#683 #684 #724)
• Disabled updatedb for JuiceFS (#727).

Changed

• Speedup listing on file store (#593).
• Upgrade KS3 SDK to 1.0.12 (#638).
• Update mtime in fallocate (#602).
• Improved performance for writing into file store (#621).
• Changed the password in Redis URL to Redis Server (#620)
• Support mixed read/write in Go/Java SDK (#647).
• Enable debug agent for sync command (#659).
• Improved stability for random write workload (#664).
• Avoid some memory copy in block cache (#668).
• disable fsync in writeback mode to be closer to local file system (#696).
• profile: show final result when interval is set to 0 (#710).

Bugfix

• Fixed stats with MySQL engine (#590).
• Fixed case insensitivity with MySQL engine(#591).
• Fixed atime of file in Java SDK(#597).
• Fixed a bug that block write when memory cache is enabled (#667).
• Fixed fd leak in block cache (#672).
• Fixed stale result for restarted transaction (#678)
• Fixed a bug under mixed write and truncate (#677).
• Fixed race condition under mixed read/write (#681).
• Fixed compatibility with Redis clones.
• Fixed data usage in read-only mode (#698).
• Fixed key leak in Redis (#694).
• Fixed a bug about collate with MySQL 5.6 (#697).
• Fixed a bug that may cause crash when writeback_cache is used (#705).
• Fixed pid of updated POSIX lock (#708).
• Added a workaround for a data loss bug in Huawei OBS SDK (#720).
• Fixed the metrics of uploaded bytes (#726).
• Fixed a leak of chunk and sustained inode in SQL engine (#728).
• Fixed a bug that may crash client (#729).

v0.15.2

JuiceFS v0.15.2 arrived 1 month after v0.14.2, with more than 60+ changes from 8 contributors (@davies, @Sandy4999 , @xiaogaozi, @yuhr123, @Suave, @zzcclp , @tangyoupeng, @chnliyong, @yuhr123), thanks to them.

This release introduced new tool to backup and restore metadata, which can also be used to migrate metadata between different meta engines, check Backup and Restore Metadata for details.

This release also improved the performance significantly for read/write heavy workload by utilizing page cache in kernel.

This release is backward-compatible with previous releases, should be safe to upgrade.

New

• Added command dump and load to backup and restore metadata (#510, #521, #529, #535, #551).
• Added an option (--read-only) to mount as read-only (#520).
• Support command auto-completion (#530, #534).
• Run benchmark in parallel (-p N) (#545).
• Support PostgreSQL as meta engine (#542).
• Added an option (--subdir) to mount a sub-directory (#550).
• Support WebDAV as an object store (#573).
• Allow enable writeback cache in kernel by -o writeback_cache (#576).
• Added a option to redirect the logging into a file in background mode (#575).

Changed

• Changed the batch size of LIST request to 1000 (some object storage may fail with 400 for larger limit) (1385587).
• Exclude log4j from Java SDK to avoid potential conflict (#501).
• Exit when unknown option found (d6a39f1).
• Report type of meta engine and storage together with usage (#504).
• Changed the default configuration of Java SDK and S3 gateway to be consistent with juicefs mount (#517).
• Keep page cache in kernel when files opened without changes (#528, #537).
• Change REDIS-URL to META-URL in docs (#552).

Bugfix

• Fixed the memory leak in B2 client (#500).
• Handle BucketAlreadyExists error for all object storage (#561).
• Fixed a bug with SQLite and PostgreSQL engine when high bit of lock owner is 1 (#588)
• Fixed updating stats for MySQL engine (#590).
• Fixed case sensitivity for MySQL engine (#591).
• Fixed potential leak for files overwritten by rename (#594, #495)

v0.14.2

JuiceFS v0.14.2 received 30+ contributions from @davies @xiaogaozi @tangyoupeng @Sandy4999 @chnliyong @yuhr123 @xyb @meilihao @frankxieke , thanks to them!

New

• Added quota for space and inodes for whole volume (#495).
• Lists all the client session in juicefs status (#491).
• Cleanup any leaked inodes in Redis with juicefs gc (#494).
• Supports sticky bits in Java SDK (#475).
• Added juicefs.umask for Java SDK (#462).
• Empty the Hadoop trash automatically (#456).

Changed

• Returns ENOSPC rather than IOError when Redis runs out of memory (#479).

Bugfix

• Allow superuser to change file mode in Java SDK (#467).
• Allow owner to change grouping of file (#470).

v0.13.1

JuiceFS v0.13.1 is a bugfix release for v0.13. We have created first release branch for 0.13, which will only have bugfix in future patch releases.

New

• Support flock for SQL engine (#422).
• Support posix lock for SQL engine (#425).
• Global user/gropu id mapping for Java SDK (#439, #447)
• Added benchmark results for different meta engine (#445).

Bugfix

• Fixed transaction conflict for SQL engine(#448).
• Fixed build on macos 11.4 (#452).
• Fixed parsing redis versions (1c945d7).
• Ignore deleted file during sync (6e06c0e).
• Fixed permission check in Lua script (used by Java SDK and S3 Gateway) (#430).
• Fixed juicefs sync in distributed mode (#424)

v0.13.0

JuiceFS v0.13 arrived 1 month after v0.12.1, with more than 80 changes from 9 contributors (@davies, @Sandy4999 , @xiaogaozi, @yuhr123, @polyrabbit, @suzaku, @tangyoupeng, @angristan, @chnliyong), thanks to them.

The biggest feature in v0.13 is using SQL database as meta engine, SQLite, MySQL and TiDB are supported right now, we will add others later. Using SQL database will be slower than using Redis, but they have better persistency and scalability than Redis, is better in the cases that data safety and number of files are more important than performance, for example, backup.

New

• Support SQL database (SQLite, MySQL and TiDB) as meta engine (#375).
• Added profile to analyze access log (#344).
• Added status to show the setting and status (#368).
• Added warmup to build cache for files/directory (#409).
• Build Java SDK for Windows (#362).
• Use multiple buckets as object store (#349).
• Collect metrics for Java SDK (#327).
• Added virtual file /.stats to show the internal metrics (#314).
• Allow build a minimized binary without S3 gateway and other object storages (#324).

Changed

• Enable checksum for Swift (a16b106).
• Added more buckets for object latency distribution (#321).
• Added a option (--no-agent) to disable debug agent (#328).
• Added internal details at the end of benchmark (#352).
• More metrics for block cache(#387).
• Speed up path resolution for Java SDK and S3 gateway using Lua script (#394).
• Restart the Redis transaction after some known failures (#397).
• Remove the limit on the number of cached blocks (#401).

Bugfix

• Fixed a bug in SetAttr to refresh new written data (429ce80).
• Fixed overflow in StatFS (abcb5c6).
• Fixed a bug when use MinIO with juicefs sync (199b4d3).
• Fixed a bug in CopyFileRange, which may affect multipart uploads and concat of Java SDK (fb611b0).
• Fixed deadlock when truncate together with read (2f8a8d9).
• Fixed stale read after truncate (226b6a7).
• Fixed downloading a directory using S3 gateway (#378).