Fix fuse mount issue on ubuntu 18.04 #7531

zpoint · 2025-10-08T10:30:34Z

This PR address the failure of smoke test: https://buildkite.com/skypilot-1/smoke-tests/builds/4096#0199bde8-a2c8-4c03-b5bb-72ee38aedbaf.

Check the Azure documentation for installing fuse2:

Latest: https://github.com/Azure/azure-storage-fuse/wiki/Blobfuse2-Installation
Ubuntu 18: https://github.com/Azure/azure-storage-fuse/wiki/Blobfuse2-Installation/a91840d19717aabbd87dd3b319e92550726a4fdf

The issue is on Ubuntu 18, fuse3 isn't in deb source and fails to install.
The blobfuse2 we downloaded from GitHub releases is compiled with fuse3.so.
So blobfuse2 will fail to run because it can't find fuse3.so even if we download it.

Approach 1: Install an older version of libfuse2 and download a blobfuse2.2 compiled with libfuse2.so

This doesn't work.

Error: unable to start pipeline [failed to mount fuse]
E1008 10:01:16.436929    6524 main.go:88] Failed to run fuse adapter: exit status 1, try to unmount /mount_empty_workdir
I1008 10:01:16.442578    6524 main.go:94] Successfully unmounted /mount_empty_workdir
command terminated with exit code 1

Our fuse-proxy needs to run in non-privileged mode and requires libfuse3's feature to mount custom fd /dev/fd/n, which libfuse2 doesn't support. Libfuse2 requires container privilege, and even with privilege set, our fuse proxy still doesn't work due to the fd mount issue. The older libfuse hardcodes mounting of /dev/fuse, which requires privilege and isn't compatible with our fuse-proxy's working logic.

Approach 2: Install libfuse3 for Ubuntu 18

This is a bit hacky but works. Use Ubuntu 20's source to install libfuse3.so for Ubuntu 18.

This PR fixes the fuse installation issue.

Reproduce:

 Step 1: Create test file
touch ~/test_file

# Step 2: Create YAML file with timestamp
TIMESTAMP=$(date +%s)
cat > /tmp/test_azure_mount.yaml << EOF
file_mounts:
  /mount_test:
    name: sky-test-${TIMESTAMP}
    source: [~/test_file]
    mode: MOUNT
    store: azure

run: |
  ls /mount_test
EOF

# Step 3: Run sky launch
sky launch -y -c t-docker-storage-15-51 --infra kubernetes --cpus 2+ --memory 4+ --image-id docker:nvidia/cuda:11.8.0-devel-ubuntu18.04 /tmp/test_azure_mount.yaml

E1008 10:01:16.436929    6524 main.go:88] Failed to run fuse adapter: exit status 1, try to unmount /mount_empty_workdir
I1008 10:01:16.442578    6524 main.go:94] Successfully unmounted /mount_empty_workdir
command terminated with exit code 1

sky.exceptions.CommandError: Command to mount failed with return code 1.
----- CMD -----
echo '
#!/usr/bin/env bash
set -e

{ [ "$(whoami)" == "root" ] && function sudo() { "$@"; } || true; }

MOUNT_PATH=$(eval echo /mount_empty_workdir)
MOUNT_BINARY=blobfuse2

# Check if path is already mounted
if findmnt -rn -T "$MOUNT_PATH" >/dev/null 2>&1; then
    echo "Path already mounted - unmounting..."
    (command -v fusermount >/dev/null 2>&1 && fusermount -uz "$MOUNT_PATH")             || (command -v fusermount3 >/dev/null 2>&1 && fusermount3 -uz "$MOUNT_PATH")             || sudo umount -l "$MOUNT_PATH" || true
    # Ensure it'"'"'s really gone (avoids races)
    for i in $(seq 1 20); do
        if ! findmnt -rn -T "$MOUNT_PATH" >/dev/null 2>&1; then break; fi
        sleep 0.2
    done
    echo "Successfully unmounted $MOUNT_PATH."
fi

# Install MOUNT_BINARY if not already installed
if [ -x "$(command -v blobfuse2)" ]; then
  echo "$MOUNT_BINARY already installed. Proceeding..."
else
  echo "Installing $MOUNT_BINARY..."
  sudo apt-get update && if sudo apt-get install -y -o Dpkg::Options::="--force-confdef" fuse3 libfuse3-dev; then   echo "fuse3 available, installing blobfuse2 via wget+dpkg";   ARCH=$(uname -m) &&   if [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then     echo "blobfuse2 is not supported on $ARCH" &&     exit 133;   else     ARCH_SUFFIX="x86_64";   fi &&   wget -nc https://github.com/Azure/azure-storage-fuse/releases/download/blobfuse2-2.2.0/blobfuse2-2.2.0-Debian-11.0.${ARCH_SUFFIX}.deb -O /tmp/blobfuse2.deb &&   sudo dpkg --install /tmp/blobfuse2.deb; else   echo "fuse3 not available, setting up Microsoft repo and installing blobfuse2";   DISTRO=$(grep "^ID=" /etc/os-release | cut -d= -f2 | tr -d '"'"'"'"'"' | tr "[:upper:]" "[:lower:]") &&   VERSION=$(grep "^VERSION_ID=" /etc/os-release | cut -d= -f2 | tr -d '"'"'"'"'"') &&   echo "Detected: $DISTRO $VERSION" &&   curl -sSL -O https://packages.microsoft.com/config/$DISTRO/$VERSION/packages-microsoft-prod.deb &&   sudo dpkg -i packages-microsoft-prod.deb &&   sudo apt-get update &&   sudo apt-get install -y   -o Dpkg::Options::="--force-confdef"   fuse libfuse-dev blobfuse2=2.2.0; fi && mkdir -p ~/.sky/blobfuse2_cache;
fi

# Check if mount path exists
if [ ! -d "$MOUNT_PATH" ]; then
  echo "Mount path $MOUNT_PATH does not exist. Creating..."
  sudo mkdir -p "$MOUNT_PATH"
  sudo chmod 777 "$MOUNT_PATH"
else
    # If not a mountpoint and contains files, clean it to satisfy SkyPilot check
    if ! findmnt -rn -T "$MOUNT_PATH" >/dev/null 2>&1; then
        if [ -n "$(ls -A "$MOUNT_PATH" 2>/dev/null)" ]; then
          echo "Cleaning non-empty mount path before mount..."
          sudo bash -lc '"'"'shopt -s dotglob nullglob; rm -rf --one-file-system -- '"'"'"$MOUNT_PATH"'"'"'/*'"'"' 2>/dev/null || true
        fi
    fi
fi
echo "Mounting $SOURCE_BUCKET to $MOUNT_PATH with $MOUNT_BINARY..."
AZURE_STORAGE_ACCOUNT=sky6356838c6a5f85a3 AZURE_STORAGE_ACCESS_KEY=TR1xs7K9Oz9Q5qCjjcSME+4RUDDD998VS1dxaMZvyoNI8tBdAXITjJ3deC49JI6rd2ean94+/jAS+AStDetdIw== $(command -v fusermount-wrapper >/dev/null 2>&1 && echo "fusermount-wrapper -d -m /mount_empty_workdir -o allow_other,default_permissions -- blobfuse2 --no-symlinks --tmp-path ~/.sky/blobfuse2_cache/sky6356838c6a5f85a3_sky-test-empty-1759909421749204_$(date +%s -d "$(uptime -s)") --container-name sky-test-empty-1759909421749204 -o nonempty --foreground {}" || echo "blobfuse2 --no-symlinks --tmp-path ~/.sky/blobfuse2_cache/sky6356838c6a5f85a3_sky-test-empty-1759909421749204_$(date +%s -d "$(uptime -s)") --container-name sky-test-empty-1759909421749204 -o allow_other -o default_permissions /mount_empty_workdir")
echo "Mounting done."
' > ~/.sky/mount_759270.sh && chmod +x ~/.sky/mount_759270.sh && bash ~/.sky/mount_759270.sh && rm ~/.sky/mount_759270.sh
----- CMD END -----
Failed to run command before rsync ~/empty-workdir -> /mount_empty_workdir. View logs: sky api logs -l sky-2025-10-08-17-56-52-340903/storage_mounts.log
Installing blobfuse2...
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Package fuse3 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'fuse3' has no installation candidate
E: Unable to locate package libfuse3-dev
fuse3 not available, setting up Microsoft repo and installing blobfuse2
Detected: ubuntu 18.04
Selecting previously unselected package packages-microsoft-prod.
(Reading database ... 18234 files and directories currently installed.)
Preparing to unpack packages-microsoft-prod.deb ...
Unpacking packages-microsoft-prod (1.0-ubuntu18.04.2) ...
Setting up packages-microsoft-prod (1.0-ubuntu18.04.2) ...
Get:1 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease [3638 B]
Get:2 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 Packages [285 kB]
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main all Packages [2395 B]
Hit:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Fetched 291 kB in 2s (119 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
fuse is already the newest version (2.9.7-1ubuntu1).
The following additional packages will be installed:
  libpcre16-3 libpcre3-dev libpcre32-3 libpcrecpp0v5 libselinux1-dev
  libsepol1-dev
The following NEW packages will be installed:
  blobfuse2 libfuse-dev libpcre16-3 libpcre3-dev libpcre32-3 libpcrecpp0v5
  libselinux1-dev libsepol1-dev
0 upgraded, 8 newly installed, 0 to remove and 36 not upgraded.
Need to get 16.8 MB of archives.
After this operation, 38.3 MB of additional disk space will be used.
Get:1 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 blobfuse2 amd64 2.2.0 [15.4 MB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsepol1-dev amd64 2.7-1ubuntu0.1 [324 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpcre16-3 amd64 2:8.39-9ubuntu0.1 [147 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpcre32-3 amd64 2:8.39-9ubuntu0.1 [138 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpcrecpp0v5 amd64 2:8.39-9ubuntu0.1 [15.3 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpcre3-dev amd64 2:8.39-9ubuntu0.1 [536 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libselinux1-dev amd64 2.7-2build2 [149 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfuse-dev amd64 2.9.7-1ubuntu1 [105 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 16.8 MB in 9s (1845 kB/s)
Selecting previously unselected package blobfuse2.
(Reading database ... 18242 files and directories currently installed.)
Preparing to unpack .../0-blobfuse2_2.2.0_amd64.deb ...
Unpacking blobfuse2 (2.2.0) ...
Selecting previously unselected package libsepol1-dev:amd64.
Preparing to unpack .../1-libsepol1-dev_2.7-1ubuntu0.1_amd64.deb ...
Unpacking libsepol1-dev:amd64 (2.7-1ubuntu0.1) ...
Selecting previously unselected package libpcre16-3:amd64.
Preparing to unpack .../2-libpcre16-3_2%3a8.39-9ubuntu0.1_amd64.deb ...
Unpacking libpcre16-3:amd64 (2:8.39-9ubuntu0.1) ...
Selecting previously unselected package libpcre32-3:amd64.
Preparing to unpack .../3-libpcre32-3_2%3a8.39-9ubuntu0.1_amd64.deb ...
Unpacking libpcre32-3:amd64 (2:8.39-9ubuntu0.1) ...
Selecting previously unselected package libpcrecpp0v5:amd64.
Preparing to unpack .../4-libpcrecpp0v5_2%3a8.39-9ubuntu0.1_amd64.deb ...
Unpacking libpcrecpp0v5:amd64 (2:8.39-9ubuntu0.1) ...
Selecting previously unselected package libpcre3-dev:amd64.
Preparing to unpack .../5-libpcre3-dev_2%3a8.39-9ubuntu0.1_amd64.deb ...
Unpacking libpcre3-dev:amd64 (2:8.39-9ubuntu0.1) ...
Selecting previously unselected package libselinux1-dev:amd64.
Preparing to unpack .../6-libselinux1-dev_2.7-2build2_amd64.deb ...
Unpacking libselinux1-dev:amd64 (2.7-2build2) ...
Selecting previously unselected package libfuse-dev.
Preparing to unpack .../7-libfuse-dev_2.9.7-1ubuntu1_amd64.deb ...
Unpacking libfuse-dev (2.9.7-1ubuntu1) ...
Setting up libsepol1-dev:amd64 (2.7-1ubuntu0.1) ...
Setting up libpcrecpp0v5:amd64 (2:8.39-9ubuntu0.1) ...
Setting up libpcre32-3:amd64 (2:8.39-9ubuntu0.1) ...
Setting up libpcre16-3:amd64 (2:8.39-9ubuntu0.1) ...
Setting up blobfuse2 (2.2.0) ...
Setting up libpcre3-dev:amd64 (2:8.39-9ubuntu0.1) ...
Setting up libselinux1-dev:amd64 (2.7-2build2) ...
Setting up libfuse-dev (2.9.7-1ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
Mount path /mount_empty_workdir does not exist. Creating...
Mounting  to /mount_empty_workdir with blobfuse2...
I1008 10:01:15.150207    6517 main.go:141] Waiting for fuse adapter running
E1008 10:01:16.443277    6517 main.go:148] Fuse adapter process exited: failure logs: I1008 10:01:15.154988    6524 main.go:79] FUSE command: [blobfuse2 --no-symlinks --tmp-path ~/.sky/blobfuse2_cache/sky6356838c6a5f85a3_sky-test-empty-1759909421749204_1759908204 --container-name sky-test-empty-1759909421749204 -o nonempty --foreground /dev/fd/9]
fuse: failed to open /dev/fuse: Operation not permitted
Error: unable to start pipeline [failed to mount fuse]
E1008 10:01:16.436929    6524 main.go:88] Failed to run fuse adapter: exit status 1, try to unmount /mount_empty_workdir
I1008 10:01:16.442578    6524 main.go:94] Successfully unmounted /mount_empty_workdir
command terminated with exit code 1

Tested (run the relevant ones):

Code formatting: install pre-commit (auto-check on commit) or bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

zpoint · 2025-10-08T10:35:30Z

sky/data/mounting_utils.py

        f'/blobfuse2-{BLOBFUSE2_VERSION}-Debian-11.0.${{ARCH_SUFFIX}}.deb '
        '-O /tmp/blobfuse2.deb && '
-        'sudo dpkg --install /tmp/blobfuse2.deb && '
+        '  sudo dpkg --install /tmp/blobfuse2.deb; '


We had a problem installing blobfuse2 on older system images like Ubuntu 18.04. This code change allows successful installation of blobfuse2.2 on older system versions.

The problem is even if we install blobfuse on Ubuntu 18, our fuse proxy doesn't seem to work for mode: MOUNT. So we should either make e-proxy support this in the future, or remove the installation code and exit with clear instructions.

zpoint · 2025-10-09T02:15:08Z

/smoke-test --remote-server --kubernetes -k test_docker_storage_mounts

zpoint · 2025-10-09T04:32:06Z

/smoke-test --kubernetes -k test_docker_storage_mounts
/smoke-test --azure -k test_docker_storage_mounts
/smoke-test --gcp -k test_docker_storage_mounts
/smoke-test --aws -k test_docker_storage_mounts

aylei

I'm confused why this does not fail before, but great work! Thanks @zpoint !

zpoint added 2 commits October 8, 2025 18:22

first commit

5bf75d0

fix

3deb80f

zpoint commented Oct 8, 2025

View reviewed changes

zpoint mentioned this pull request Oct 8, 2025

[Tests] Don't Include Azure Mount if 18.04 #7518

Merged

5 tasks

zpoint requested review from Michaelvll and aylei October 8, 2025 10:56

zpoint added 3 commits October 8, 2025 18:59

remove fallback

03e0642

Merge branch 'master' into dev/zeping/fix_mount_issue

9690548

skip azure if not support

a78c1f9

force install fuse3 for ubuntu 18

9215683

zpoint changed the title ~~Fix fuse mount issue~~ Fix fuse mount issue on ubuntu 18.04 Oct 9, 2025

aylei approved these changes Oct 9, 2025

View reviewed changes

zpoint merged commit ab29876 into skypilot-org:master Oct 10, 2025
32 of 34 checks passed

zpoint mentioned this pull request Oct 10, 2025

test_kubernetes_storage_mounts flaky on nightly build #7410

Closed

kevinmingtarja mentioned this pull request Nov 1, 2025

[Storage] Fix Azure mount install cmd to not reinstall fuse3 when fusermount-shim is used #7818

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix fuse mount issue on ubuntu 18.04 #7531

Fix fuse mount issue on ubuntu 18.04 #7531

zpoint commented Oct 8, 2025 •

edited

Loading

Uh oh!

zpoint Oct 8, 2025

Uh oh!

zpoint Oct 8, 2025 •

edited

Loading

Uh oh!

zpoint commented Oct 9, 2025

Uh oh!

zpoint commented Oct 9, 2025

Uh oh!

aylei left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix fuse mount issue on ubuntu 18.04 #7531

Fix fuse mount issue on ubuntu 18.04 #7531

Conversation

zpoint commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approach 1: Install an older version of libfuse2 and download a blobfuse2.2 compiled with libfuse2.so

Approach 2: Install libfuse3 for Ubuntu 18

Reproduce:

Uh oh!

zpoint Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

zpoint Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zpoint commented Oct 9, 2025

Uh oh!

zpoint commented Oct 9, 2025

Uh oh!

aylei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zpoint commented Oct 8, 2025 •

edited

Loading

zpoint Oct 8, 2025 •

edited

Loading