Longhorn v1.6.1

Longhorn v1.6.1 Release Notes

Longhorn 1.6.1 introduces several improvements and bug fixes that are intended to improve system quality, resilience, and stability.

The Longhorn team appreciates your contributions and expects to receive feedback regarding this release.

Note

For more information about release-related terminology, see Releases.

Installation

Important

Ensure that your cluster is running Kubernetes v1.21 or later before installing Longhorn v1.6.1.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Important

Ensure that your cluster is running Kubernetes v1.21 or later before upgrading from Longhorn v1.5.x or v1.6.x to v1.6.1.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Deprecation & Incompatibilities

For information about important changes, including feature incompatibility, deprecation, and removal, see Important Notes in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Improvement

• [BACKPORT][v1.6.1][IMPROVEMENT] Add dmsetup and dmcrypt utilities check in environment check script 8234 - @derekbit @chriscchien
• [BACKPORT][v1.6.1][IMPROVEMENT] Upgrade support bundle kit version to v0.0.36 8162 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Cannot read/write to block volume when the container is run as non-root 8122 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.1][IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7971 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7973 - @votdev @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7886 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 8000 - @ChanYiLin @roger-ryao

Bug

• [BUG][v1.6.1-rc3] test_backup_lock_creation_during_deletion failed with *.lck type 1 acquisition 8269 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] Replica rebuild failed 8258 - @shuo-wu @roger-ryao
• [BACKPORT][v1.6.1][BUG] potential risk to unmap a negative number 8237 - @Vicente-Cheng @roger-ryao
• [BACKPORT][v1.6.1][BUG] Can't use longhorn with Generic ephemeral volumes 8213 - @ejweber @chriscchien
• [BUG] [v1.6.1-rc1] v2 volume replica offline rebuilding fail 8187 - @shuo-wu @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn api-server PUT request rate 8152 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Use config map to update default-replica-count won't apply to default-replica-count.definition.default if the value equal to current default-replica-count.value 8134 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][BUG] Failed to restore a backup to file by the scripts/restore-backup-to-file.sh with a CIFS backup target. 8128 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.1][BUG] Exporting data from the existing replicas has issues in 1.6.0 8096 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] longhorn manager pod fails to start in container-based K3s 7847 - @ChanYiLin @khushboo-rancher @chriscchien
• [BACKPORT][v1.6.1][BUG] no Pending workload pods for volume xxx to be mounted 8081 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock is possible in v1.6.0 instance manager 7920 - @roger-ryao
• [BACKPORT][v1.6.1][BUG][1.6.0] ENGINE v2 : disk /dev/xxxx is already used by AIO bdev disk-x 8133 - @derekbit @chriscchien
• [BACKPORT][v1.6.1][BUG] A replica may be incorrectly scheduled to a node with an existing failed replica 8044 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7870 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volume cannot attach because of the leftover non-empty volume.status.PendingNodeID after upgrading Longhorn 7995 - @james-munson @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7885 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] LH manager reboots due to the webhook is not ready 8037 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] BackingImage does not download URL correctly in some situation 7987 - @votdev @yangchiu
• [BACKPORT][v1.6.1][BUG] Executing fstrim while rebuilding causes IO errors 7868 - @yangchiu @ejweber
• [BACKPORT][v1.6.1][BUG] The feature of auto remount read only volume not work on a single node cluster. 7846 - @yangchiu @ChanYiLin
• [BACKPORT][v1.6.1][BUG] Missed NodeStageVolume after reboot leads to CreateContainerError 8012 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] persistence.removeSnapshotsDuringFilesystemTrim Helm variable is unreferenced 7952 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG][v1.5.4-rc4] Test case test_backuptarget_available_during_engine_image_not_ready failed to wait for backup target available 8055 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Add Snapshot Maximum Count to the Settings 7979 - @FrankYang0529 @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7899 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] The activated DR volume do not contain the latest data. 7946 - @shuo-wu @roger-ryao
• [BACKPORT][v1.6.1][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8013 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.6.1][BUG] Create backup failed: failed lock lock-*.lck type 1 acquisition 7875 - @yangchiu @ChanYiLin @chriscchien
• [BUG] Fix errors in questions.yaml 6392 - @james-munson @chriscchien

Misc

• [BACKPORT][v1.6.1][REFACTOR] move mount point check function to common lib 8124 - @ChanYiLin @chriscchien
• [TASK] Fix updating patch digest dependencies (v1.6.x) 8107 - @mantissahz

Contributors

• @ChanYiLin
• @FrankYang0529
• @PhanLe1010
• @Vicente-Cheng
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @james-munson
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @shuo-wu
• @votdev
• @yangchiu

Longhorn v1.6.1-rc3

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Improvement

• [BACKPORT][v1.6.1][IMPROVEMENT] Add dmsetup and dmcrypt utilities check in environment check script 8234 - @derekbit @chriscchien
• [BACKPORT][v1.6.1][IMPROVEMENT] Upgrade support bundle kit version to v0.0.36 8162 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Cannot read/write to block volume when the container is run as non-root 8122 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.1][IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7971 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7973 - @votdev @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7886 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 8000 - @ChanYiLin @roger-ryao

Bug

• [BACKPORT][v1.6.1][BUG] potential risk to unmap a negative number 8237 - @Vicente-Cheng @roger-ryao
• [BACKPORT][v1.6.1][BUG] Replica rebuild failed 8258 - @shuo-wu @roger-ryao
• [BACKPORT][v1.6.1][BUG] Can't use longhorn with Generic ephemeral volumes 8213 - @ejweber @chriscchien
• [BUG] [v1.6.1-rc1] v2 volume replica offline rebuilding fail 8187 - @shuo-wu @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn api-server PUT request rate 8152 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Use config map to update default-replica-count won't apply to default-replica-count.definition.default if the value equal to current default-replica-count.value 8134 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][BUG] Failed to restore a backup to file by the scripts/restore-backup-to-file.sh with a CIFS backup target. 8128 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.1][BUG] Exporting data from the existing replicas has issues in 1.6.0 8096 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] longhorn manager pod fails to start in container-based K3s 7847 - @ChanYiLin @khushboo-rancher @chriscchien
• [BACKPORT][v1.6.1][BUG] no Pending workload pods for volume xxx to be mounted 8081 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock is possible in v1.6.0 instance manager 7920 - @roger-ryao
• [BACKPORT][v1.6.1][BUG][1.6.0] ENGINE v2 : disk /dev/xxxx is already used by AIO bdev disk-x 8133 - @derekbit @chriscchien
• [BACKPORT][v1.6.1][BUG] A replica may be incorrectly scheduled to a node with an existing failed replica 8044 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7870 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volume cannot attach because of the leftover non-empty volume.status.PendingNodeID after upgrading Longhorn 7995 - @james-munson @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7885 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] LH manager reboots due to the webhook is not ready 8037 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] BackingImage does not download URL correctly in some situation 7987 - @votdev @yangchiu
• [BACKPORT][v1.6.1][BUG] Executing fstrim while rebuilding causes IO errors 7868 - @yangchiu @ejweber
• [BACKPORT][v1.6.1][BUG] The feature of auto remount read only volume not work on a single node cluster. 7846 - @yangchiu @ChanYiLin
• [BACKPORT][v1.6.1][BUG] Missed NodeStageVolume after reboot leads to CreateContainerError 8012 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] persistence.removeSnapshotsDuringFilesystemTrim Helm variable is unreferenced 7952 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG][v1.5.4-rc4] Test case test_backuptarget_available_during_engine_image_not_ready failed to wait for backup target available 8055 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Add Snapshot Maximum Count to the Settings 7979 - @FrankYang0529 @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7899 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] The activated DR volume do not contain the latest data. 7946 - @shuo-wu @roger-ryao
• [BACKPORT][v1.6.1][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8013 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.6.1][BUG] Create backup failed: failed lock lock-*.lck type 1 acquisition 7875 - @yangchiu @ChanYiLin @chriscchien
• [BUG] Fix errors in questions.yaml 6392 - @james-munson @chriscchien

Misc

• [BACKPORT][v1.6.1][REFACTOR] move mount point check function to common lib 8124 - @ChanYiLin @chriscchien
• [TASK] Fix updating patch digest dependencies (v1.6.x) 8107 - @mantissahz

Contributors

• @ChanYiLin
• @FrankYang0529
• @PhanLe1010
• @Vicente-Cheng
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @james-munson
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @shuo-wu
• @votdev
• @yangchiu

Longhorn v1.6.1-rc2

Warning

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Improvement

• [BACKPORT][v1.6.1][IMPROVEMENT] Upgrade support bundle kit version to v0.0.36 8162 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Allow users to request backup volume update 7984 - @mantissahz
• [BACKPORT][v1.6.1][IMPROVEMENT] Cannot read/write to block volume when the container is run as non-root 8122 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.1][IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7971 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7973 - @votdev @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7886 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 8000 - @ChanYiLin @roger-ryao

Bug

• [BUG] [v1.6.1-rc1] v2 volume replica offline rebuilding fail 8187 - @shuo-wu @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn api-server PUT request rate 8152 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Use config map to update default-replica-count won't apply to default-replica-count.definition.default if the value equal to current default-replica-count.value 8134 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][BUG] Failed to restore a backup to file by the scripts/restore-backup-to-file.sh with a CIFS backup target. 8128 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.1][BUG] Exporting data from the existing replicas has issues in 1.6.0 8096 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] longhorn manager pod fails to start in container-based K3s 7847 - @ChanYiLin @khushboo-rancher @chriscchien
• [BACKPORT][v1.6.1][BUG] no Pending workload pods for volume xxx to be mounted 8081 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock is possible in v1.6.0 instance manager 7920 - @roger-ryao
• [BACKPORT][v1.6.1][BUG] metric longhorn_volume_robustness did not reflect detached volume 8140 - @c3y1huang
• [BACKPORT][v1.6.1][BUG][1.6.0] ENGINE v2 : disk /dev/xxxx is already used by AIO bdev disk-x 8133 - @derekbit @chriscchien
• [BACKPORT][v1.6.1][BUG] A replica may be incorrectly scheduled to a node with an existing failed replica 8044 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7870 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volume cannot attach because of the leftover non-empty volume.status.PendingNodeID after upgrading Longhorn 7995 - @james-munson @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7885 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] LH manager reboots due to the webhook is not ready 8037 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] BackingImage does not download URL correctly in some situation 7987 - @votdev @yangchiu
• [BACKPORT][v1.6.1][BUG] Executing fstrim while rebuilding causes IO errors 7868 - @yangchiu @ejweber
• [BACKPORT][v1.6.1][BUG] The feature of auto remount read only volume not work on a single node cluster. 7846 - @yangchiu @ChanYiLin
• [BACKPORT][v1.6.1][BUG] Missed NodeStageVolume after reboot leads to CreateContainerError 8012 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] persistence.removeSnapshotsDuringFilesystemTrim Helm variable is unreferenced 7952 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG][v1.5.4-rc4] Test case test_backuptarget_available_during_engine_image_not_ready failed to wait for backup target available 8055 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Add Snapshot Maximum Count to the Settings 7979 - @FrankYang0529 @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7899 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] The activated DR volume do not contain the latest data. 7946 - @shuo-wu @roger-ryao
• [BACKPORT][v1.6.1][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8013 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.6.1][BUG] Create backup failed: failed lock lock-*.lck type 1 acquisition 7875 - @yangchiu @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG][v1.5.4-rc1] Recurring job failed to create/delete backups after node reboot 7855 - @james-munson
• [BUG] Fix errors in questions.yaml 6392 - @james-munson @chriscchien

Misc

• [BACKPORT][v1.6.1][REFACTOR] move mount point check function to common lib 8124 - @ChanYiLin @chriscchien
• [TASK] Fix updating patch digest dependencies (v1.6.x) 8107 - @mantissahz

Contributors

• @ChanYiLin
• @FrankYang0529
• @PhanLe1010
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @james-munson
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @shuo-wu
• @votdev
• @yangchiu

Longhorn v1.6.1-rc1

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Improvement

• [BACKPORT][v1.6.1][IMPROVEMENT] Upgrade support bundle kit version to v0.0.36 8162 - @c3y1huang @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Allow users to request backup volume update 7984 - @mantissahz
• [BACKPORT][v1.6.1][IMPROVEMENT] Cannot read/write to block volume when the container is run as non-root 8122 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.6.1][IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7971 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7973 - @votdev @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7886 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 8000 - @ChanYiLin @roger-ryao

Bug

• [BACKPORT][v1.6.1][BUG] Exporting data from the existing replicas has issues in 1.6.0 8096 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn api-server PUT request rate 8152 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] no Pending workload pods for volume xxx to be mounted 8081 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock is possible in v1.6.0 instance manager 7920 - @roger-ryao
• [BACKPORT][v1.6.1][BUG] metric longhorn_volume_robustness did not reflect detached volume 8140 - @c3y1huang
• [BACKPORT][v1.6.1][BUG][1.6.0] ENGINE v2 : disk /dev/xxxx is already used by AIO bdev disk-x 8133 - @derekbit @chriscchien
• [BACKPORT][v1.6.1][BUG] A replica may be incorrectly scheduled to a node with an existing failed replica 8044 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7870 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volume cannot attach because of the leftover non-empty volume.status.PendingNodeID after upgrading Longhorn 7995 - @james-munson @chriscchien
• [BACKPORT][v1.6.1][BUG] Use config map to update default-replica-count won't apply to default-replica-count.definition.default if the value equal to current default-replica-count.value 8134 - @james-munson @roger-ryao
• [BACKPORT][v1.6.1][BUG] Failed to restore a backup to file by the scripts/restore-backup-to-file.sh with a CIFS backup target. 8128 - @mantissahz @roger-ryao
• [BACKPORT][v1.6.1][BUG] longhorn manager pod fails to start in container-based K3s 7847 - @ChanYiLin @khushboo-rancher @chriscchien
• [BACKPORT][v1.6.1][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7885 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] LH manager reboots due to the webhook is not ready 8037 - @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG] BackingImage does not download URL correctly in some situation 7987 - @votdev @yangchiu
• [BACKPORT][v1.6.1][BUG] Executing fstrim while rebuilding causes IO errors 7868 - @yangchiu @ejweber
• [BACKPORT][v1.6.1][BUG] The feature of auto remount read only volume not work on a single node cluster. 7846 - @yangchiu @ChanYiLin
• [BACKPORT][v1.6.1][BUG] Missed NodeStageVolume after reboot leads to CreateContainerError 8012 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG] persistence.removeSnapshotsDuringFilesystemTrim Helm variable is unreferenced 7952 - @ejweber @chriscchien
• [BACKPORT][v1.6.1][BUG][v1.5.4-rc4] Test case test_backuptarget_available_during_engine_image_not_ready failed to wait for backup target available 8055 - @c3y1huang @chriscchien
• [BACKPORT][v1.6.1][BUG] Add Snapshot Maximum Count to the Settings 7979 - @FrankYang0529 @roger-ryao
• [BACKPORT][v1.6.1][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7899 - @ejweber @roger-ryao
• [BACKPORT][v1.6.1][BUG] The activated DR volume do not contain the latest data. 7946 - @shuo-wu @roger-ryao
• [BACKPORT][v1.6.1][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8013 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.6.1][BUG] Create backup failed: failed lock lock-*.lck type 1 acquisition 7875 - @yangchiu @ChanYiLin @chriscchien
• [BACKPORT][v1.6.1][BUG][v1.5.4-rc1] Recurring job failed to create/delete backups after node reboot 7855 - @james-munson
• [BUG] Fix errors in questions.yaml 6392 - @james-munson @chriscchien

Misc

• [BACKPORT][v1.6.1][REFACTOR] move mount point check function to common lib 8124 - @ChanYiLin @chriscchien
• [TASK] Fix updating patch digest dependencies (v1.6.x) 8107 - @mantissahz

Contributors

• @ChanYiLin
• @FrankYang0529
• @PhanLe1010
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @james-munson
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @shuo-wu
• @votdev
• @yangchiu

Longhorn v1.5.4

Longhorn v1.5.4 Release Notes

This latest stable version of Longhorn 1.5 introduces several improvements and bug fixes that are intended to improve system quality, resilience, and stability.

The Longhorn team appreciates your contributions and anticipates receiving feedback regarding this release.

Note:
For more information about release-related terminology, see Releases.

Installation

Ensure that your cluster is running Kubernetes v1.21 or later before installing Longhorn v1.5.4.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Ensure that your cluster is running Kubernetes v1.21 or later before upgrading from Longhorn v1.4.x to v1.5.4.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Deprecation & Incompatibilities

For information about important changes, including feature incompatibility, deprecation, and removal, see Important Notes in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7981 - @votdev @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 7999 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse
• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7933 - @ejweber @chriscchien
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BACKPORT][v1.5.4][BUG][v1.5.4-rc4] Test case test_backuptarget_available_during_engine_image_not_ready failed to wait for backup target available 8054 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] The activated DR volume do not contain the latest data. 7947 - @shuo-wu @roger-ryao
• [BUG][v1.5.x] DR volume unable to be activated if the latest backup's been deleted 7997 - @yangchiu @shuo-wu
• [BUG] Backup related test cases failed 7989 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8015 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.5.4][BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7186 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock is possible in v1.6.0 instance manager 7941 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] BackingImage does not download URL correctly in some situation 7986 - @yangchiu
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7901 - @yangchiu @ejweber
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @chriscchien @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space [7306](https://github.com/longh...

Longhorn v1.5.4-rc4

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7981 - @votdev @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 7999 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7974 - @votdev
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse
• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7933 - @ejweber @chriscchien
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BUG][v1.5.x] DR volume unable to be activated if the latest backup's been deleted 7997 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8015 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.5.4][BUG] The activated DR volume do not contain the latest data. 7947 - @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7186 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock is possible in v1.6.0 instance manager 7941 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber @chriscchien
• [BUG] Backup related test cases failed 7989 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG] BackingImage does not download URL correctly in some situation 7986 - @yangchiu
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7901 - @yangchiu @ejweber
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @chriscchien @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @mantissahz @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims success when kubectl fails. 7216 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backi...

Longhorn v1.5.4-rc3

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Improvement

• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse
• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7981 - @votdev @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7974 - @votdev
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7969 - @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7933 - @ejweber @chriscchien
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BUG][v1.5.x] Recurring job fails to create backup when volume detached 7937 - @mantissahz @c3y1huang
• [BUG] Backup related test cases failed 7989 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG] The activated DR volume do not contain the latest data. 7947 - @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] BackingImage does not download URL correctly in some situation 7986 - @yangchiu
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock is possible in v1.6.0 instance manager 7941 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7901 - @yangchiu @ejweber
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @chriscchien @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @mantissahz @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims success when kubectl fails. 7216 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 7701 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7426 - @ejweber @roger-ryao
• [BA...

Longhorn v1.5.4-rc2

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Feature

• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG][v1.5.4-rc1] Recurring job failed to create/delete backups after node reboot 7856 - @james-munson
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Create backup failed: failed lock lock-*.lck type 1 acquisition 7876 - @yangchiu @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @mantissahz @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims success when kubectl fails. 7216 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 7701 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7426 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Volume could not be remounted after engine process killed 7772 - @ChanYiLin @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] Enabling replica-auto-balance tries to replicate to disabled nodes causing lots of errors in the logs and in the UI 7275 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] allow-collecting-longhorn-usage-metrics setting is missing from chart settings 7250 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][BUG] When disabling revision counter, salvaging a faulty volume not work as expected 7732 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] During volume live engine upgrade, delete replica with old engine image will make volume degraded forever 7334 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Uninstallation job stuck forever if the MutatingWebhookConfigurations or ValidatingWebhookConfigurations already deleted 7658 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][BUG] Volume encryption doesn't work on Amazon Linux 2 [7165](https://github.com/longhorn/longhorn/issue...

Longhorn v1.6.0

Longhorn v1.6.0 Release Notes

This latest version of Longhorn introduces several features, enhancements, and bug fixes that are intended to improve system quality and the overall user experience. Highlights include new V2 Data Engine features, platform-agnostic deployment, node maintenance, and improvements to stability, performance, and resilience.

The Longhorn team appreciates your contributions and anticipates receiving feedback regarding this release.

Note:
For more information about release-related terminology, see Releases.

Primary Highlights

New V2 Data Engine Features

Although the V2 Data Engine is still considered a preview feature in this release, the core functions have been significantly enhanced. For example, you can now seamlessly perform volume backup and restore operations between the V1 and V2 Data Engines, paving the way for volume migration between the two data engines in the future.

• Volume Snapshot and Revert
• Volume Backup and Restore
• Separate Data Plane for v1 and v2 Data Engines
• ARM64 Support

The Longhorn team will continue to develop features for the V1 Data Engine and to prepare the V2 Data Engine for use in all types of environments.

Platform-Agnostic Deployment

Longhorn is designed to seamlessly operate on general-purpose Linux distributions, and on certain container-optimized systems such as SLE Micro. In response to numerous requests, v1.6.0 was enhanced to allow installation of Longhorn components on Talos, which is a secure, immutable, and minimal Kubernetes OS. v1.6.0 also includes OKD support, which was contributed by community member @ArthurVardevanyan.

• Talos Support
• OKD (OpenShift Origin) Support

The Longhorn team is committed to making Longhorn an adaptive storage solution and anticipates receiving feedback regarding your preferred platforms.

Space Efficiency

Starting with v1.6.0, Longhorn allows you to configure the maximum snapshot count and the maximum aggregate snapshot size for all volumes and for specific volumes. Both settings, whether applied globally or individually, aid in space estimation and management. Earlier Longhorn versions do not provide mechanisms for controlling or predicting the quantity and size of volume snapshots.

• Snapshot Space Management

GitOps Friendly

Longhorn has been validated with popular GitOps solutions, including Flux, Argo CD, and Fleet. Future releases will include enhancements that further solidify Longhorn's status as a GitOps-aware storage solution.

Data Protection

Longhorn now supports block volume encryption, which is particularly beneficial in virtualization use cases such as Harvester and KubeVirt.

Node Maintenance

v1.6.0 includes two new node drain policy options: Block For Eviction and Block For Eviction If Contains Last Replica. Both options allow automatic eviction and relocation of healthy replicas from draining nodes (before the nodes are cordoned).

The Longhorn team recommends enabling these options only during planned maintenance to minimize impact on data movement. For more information about the advantages and disavantages of all options, see Node Drain Policy Recommendations in the Longhorn documentation.

Backing Image Management

Longhorn now allows you to create and restore backups of backing images, which can streamline the management of backing images across clusters. This feature is particularly beneficial in virtualization use cases such as Harvester and KubeVirt.

Installation

Ensure that your cluster is running Kubernetes v1.21 or later before installing Longhorn v1.6.0.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Ensure that your cluster is running Kubernetes v1.21 or later before upgrading from Longhorn v1.5.x to v1.6.0.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Deprecation & Incompatibilities

For information about important changes, including feature incompatibility, deprecation, and removal, see Important Notes in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Highlights

• [FEATURE] Longhorn snapshot space management 6563 - @FrankYang0529 @yangchiu
• [FEATURE] v2 data engine volume snapshot and revert 6137 - @shuo-wu @roger-ryao
• [FEATURE] Support eventual danger zone setting update 7173 - @mantissahz @chriscchien
• [FEATURE] Engine upgrade enforcement 5842 - @yangchiu @mantissahz @c3y1huang
• [FEATURE] Selective V2 Data Engine Activation 7015 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz @chriscchien
• [FEATURE] v2 volume supports volume backup/restore 6138 - @yangchiu @derekbit
• [IMPROVEMENT] Remove or Change Helm pre-upgrade hook to support ArgoCD 6415 - @mantissahz
• [FEATURE] Restore BackingImage for BackupVolume in a new cluster 4165 - @ChanYiLin @roger-ryao
• [FEATURE] Talos support 3161 - @yangchiu @c3y1huang
• [FEATURE] Support v2 volume on ARM64 platform 6021 - @derekbit @chriscchien @roger-ryao
• [IMPROVEMENT] Add a new setting that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber @chriscchien
• [FEATURE] Add linear dm device on the top of v2 volume 7357 - @derekbit @chriscchien
• [FEATURE] Support Encryption for VolumeMode Block 4883 - @derekbit @roger-ryao
• [TASK] Add install/upgrade longhorn by gitops (flux) pipeline 6343 - @yangchiu
• [FEATURE] OKD/Openshift support 1831 - @mantissahz @ArthurVardevanyan @roger-ryao

Features

• [UI][FEATURE] Longhorn snapshot space management 7522 - @yangchiu @scures
• [FEATURE] RWX volume supports different NFS version (4.2) and mount options 7638 - @james-munson
• [FEATURE] Introduce upgradeVersionCheck to decide version upgrade enforcement 7539 - @mantissahz @chriscchien
• [FEATURE] v2 volume replica management 5420 - @DamiaSan
• [FEATURE] Update nfs-genesha to 5.x for share manager 6000 - @james-munson @chriscchien
• [FEATURE] Allow to set mount options for storageclass via values.yaml in helm chart 7351 - @ChanYiLin @chriscchien
• [FEATURE] Flush on-the-fly IOs in the queue before snapshotting 5648 - @DamiaSan
• [FEATURE] Update base image of Longhorn components to BCI 15.5 6206 - @nitendra-suse
• [FEATURE] Customize MaxRecurringJobRetain 5713 - @mantissahz @chriscchien
• [FEATURE] Replica rebuild over SPDK 5216 - @shuo-wu @DamiaSan
• [FEATURE] Allow kubectl drain to stop manually attached volumes 6978 - @ChanYiLin @chriscchien
• [FEATURE] Single Node Disk affinity 3823 - @ejweber @roger-ryao
• [FEATURE] Storage network support for Multus v4.0 thick-plugin 5048 - @c3y1huang @chriscchien
• [FEATURE] Add disk status prometheus metrics 6858 - @c3y1huang @chriscchien
• [FEATURE] Add a brand new/empty bdev with WriteOnly mode to the RAID1 bdev 5865 - @DamiaSan
• [FEATURE] Add ...

Longhorn v1.5.4-rc1

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Feature

• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien

Bug

• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @c3y1huang
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz
• [BUG] longhorn manager pod fails to start in container-based K3s 5693 - @ChanYiLin @andrewd-zededa
• [BACKPORT][v1.5.4][BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 7701 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit
• [BACKPORT][v1.5.4][BUG] When disabling revision counter, salvaging a faulty volume not work as expected 7732 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] During volume live engine upgrade, delete replica with old engine image will make volume degraded forever 7334 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Uninstallation job stuck forever if the MutatingWebhookConfigurations or ValidatingWebhookConfigurations already deleted 7658 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber
• [BACKPORT][v1.5.4][BUG] Volume encryption doesn't work on Amazon Linux 2 7165 - @derekbit @chriscchien
• [BACKPORT][v1.5.4][BUG] Rancher cannot import longhorn 1.5 charts due to "error converting YAML to JSON: yaml: line 699: did not find expected key" 7776 - @mantissahz @PhanLe1010
• [BACKPORT][v1.5.4][BUG] allow-collecting-longhorn-usage-metrics setting is missing from chart settings 7250 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7186 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Delete kubernetes node did not remove node.longhorn.io 7538 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume could not be remounted after engine process killed 7772 - @ChanYiLin @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] Enabling replica-auto-balance tries to replicate to disabled nodes causing lots of errors in the logs and in the UI 7275 - @yangchiu @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @c3y1huang
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG] backingimage download server error 7381 - @scures @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn-manager does not deploy CSI driver when integrated with linkerd service mesh 7391 - @yangchiu @mantissahz
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7426 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Helm2 install error: 'lookup' function not defined in validate-psp-install.yaml 7435 - @roger-ryao
• [BACKPORT][v1.5.4][BUG] Warning events are being spammed by Longhorn - CRD 7309 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Persistent volume is not ready for workloads 7314 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Download backing image failed with HTTP 502 error if Storage Network configured 7239 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Errors found by static checker in volume controller 7269 - @m-ildefons
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims...