Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

❓ [Question] Jetson AGX Orin build and install torch_tensorrt wheel file Failed #3331

Open
breknddone opened this issue Dec 18, 2024 · 5 comments
Open

❓ [Question] Jetson AGX Orin build and install torch_tensorrt wheel file Failed #3331

breknddone opened this issue Dec 18, 2024 · 5 comments
Labels
question Further information is requested

Comments

@breknddone
Copy link

❓ Question

I follow this tutorial to install Torch-TensorRT, but in the last step:

cuda_version=$(nvcc --version | grep Cuda | grep release | cut -d ',' -f 2 | sed -e 's/ release //g')
export TORCH_INSTALL_PATH=$(python -c "import torch, os; print(os.path.dirname(torch.__file__))")
export SITE_PACKAGE_PATH=${TORCH_INSTALL_PATH::-6}
export CUDA_HOME=/usr/local/cuda-${cuda_version}/
# replace the MODULE.bazel with the jetpack one
cat toolchains/jp_workspaces/MODULE.bazel.tmpl | envsubst > MODULE.bazel
# build and install torch_tensorrt wheel file
python setup.py --use-cxx11-abi install --user

some errors happened:

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
    LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
    PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/core/conversion/var/BUILD:20:11: Compiling core/conversion/var/Var.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/58/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/core/DeprecatedTypePropertiesRegistry.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:59:11: Compiling core/conversion/converters/NodeConverterRegistry.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/57/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/ops/cudnn_batch_norm_ops.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:39:11: Compiling core/conversion/converters/converter_util.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/56/execroot/_main/external/_main~_repo_rules~libtorch/include/ATen/ops/native_dropout_backward_cpu_dispatch.h (???????)
Target //:libtorchtrt failed to build
INFO: Elapsed time: 1000.299s, Critical Path: 574.06s
INFO: 7984 processes: 7938 internal, 46 linux-sandbox.
ERROR: Build did NOT complete successfully

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • PyTorch Version (e.g., 1.0): 2.5.0
  • CPU Architecture: arm64(Jetson AGX Orin)
  • OS (e.g., Linux): Linux
  • How you installed PyTorch: pip
  • Build command you used (if compiling from source): python setup.py --use-cxx11-abi install --user
  • Are you using local sources or building from archives: building from archives
  • Python version: 3.10.15
  • CUDA version: 12.6
  • GPU models and configuration: -
  • Any other relevant information: Install torch_tensorrt in the model's anaconda virtual environment

Additional context

It seems a I/O exception.But Jetson still has 11GB of space.please help me!thanks!!!!
@breknddone breknddone added the question Further information is requested label Dec 18, 2024
@narendasan
Copy link
Collaborator

Can you change the bazel build command in setup.py to add --sandbox-debug and --verbose_failures and share the output? This message does not convey that much information

@breknddone
Copy link
Author

Can you change the bazel build command in setup.py to add --sandbox-debug and --verbose_failures and share the output? This message does not convey that much information

sure.

DEBUG: Sandbox debug output for CppCompile 
//core/conversion/tensorcontainer:tensorcontainer:
1734547440.188853798: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734547440.188938086: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734547440.188945350: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734547440.189515846: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 13275
1734547440.190429253: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734547440.190705893: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734547440.193842786: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/_hermetic_tmp -> /tmp
1734547440.199941085: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734547440.199974141: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.199992253: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734547440.200001981: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.200251197: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734547440.200269181: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734547440.200277373: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547440.200297469: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734547440.200308765: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547440.200315773: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734547440.200322333: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734547440.200336125: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734547440.200346397: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734547440.200355933: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734547440.200364477: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734547440.200379421: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734547440.200387037: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734547440.200395357: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734547440.200404892: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734547440.200475804: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734547440.200488412: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734547440.200494492: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547440.200501532: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734547440.200507708: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734547440.200513532: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734547440.200520348: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734547440.200526172: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734547440.200532700: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734547440.200540124: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734547440.200549500: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734547440.200556764: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734547440.200563388: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734547440.200615580: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734547440.200658108: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734547440.200669852: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734547440.200697052: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734547440.200705468: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734547440.200711548: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734547440.200719836: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734547440.200728252: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734547440.200735516: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734547440.200742172: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734547440.200752508: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/bszk
1734547440.200776476: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734547440.200784572: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547440.200791356: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547440.200798428: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.200808636: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547440.200815292: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main
1734547440.200945436: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734547440.201218492: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734547597.851306145: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x00
1734547597.852080283: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 0
1734547597.853295897: src/main/tools/linux-sandbox.cc:243: child exited normally with code 0

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/39/debug.out -- /bin/sh -i)
DEBUG: Sandbox debug output for CppCompile //core/ir:ir:
1734547666.568098834: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734547666.568173129: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734547666.568179816: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734547666.568852633: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 13473
1734547666.572746858: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734547666.572907127: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734547666.573341860: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/_hermetic_tmp -> /tmp
1734547666.573399581: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734547666.573421658: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.573440472: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734547666.573451735: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.573617731: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734547666.573633249: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734547666.573641088: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547666.573661630: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734547666.573672317: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547666.573680380: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734547666.573686715: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734547666.573699865: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734547666.573710360: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734547666.573720823: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734547666.573729430: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734547666.573743604: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734547666.573751603: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734547666.573759186: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734547666.573767537: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734547666.573834313: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734547666.573846952: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734547666.573852807: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547666.573861798: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734547666.573869829: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734547666.573877348: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734547666.573884803: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734547666.573891043: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734547666.573900098: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734547666.573908577: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734547666.573918015: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734547666.573927998: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734547666.573938717: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734547666.573989751: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734547666.574002453: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734547666.574012948: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734547666.574039409: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734547666.574047952: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734547666.574054671: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734547666.574063694: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734547666.574073197: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734547666.574080940: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734547666.574089643: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734547666.574101226: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/bszk
1734547666.574123239: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734547666.574131366: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547666.574138469: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547666.574144613: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.574155491: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547666.574162786: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main
1734547666.574342157: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734547666.574685476: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734547783.023817798: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x00
1734547783.023855490: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 0
1734547783.024807844: src/main/tools/linux-sandbox.cc:243: child exited normally with code 0

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/47/debug.out -- /bin/sh -i)
DEBUG: Sandbox debug output for CppCompile //core/ir:ir:
1734547645.822595679: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1734547645.822668214: src/main/tools/linux-sandbox.cc:165: Netns is 0
1734547645.822675189: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1734547645.823208308: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 13464
1734547645.823293929: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1734547645.823395485: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1734547645.823736787: src/main/tools/linux-sandbox-pid1.cc:293: bind mount: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -> /tmp
1734547645.823784685: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1734547645.823803563: src/main/tools/linux-sandbox-pid1.cc:311: writable: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.823821352: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1734547645.823832647: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.823980661: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1734547645.823995507: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1734547645.824004946: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547645.824025039: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1734547645.824034862: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1734547645.824042445: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1734547645.824049004: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1734547645.824061579: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/selinux
1734547645.824074857: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1734547645.824083368: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1734547645.824091783: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1734547645.824106341: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1734547645.824114308: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1734547645.824121955: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1734547645.824130594: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1734547645.824195066: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1734547645.824206905: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1734547645.824213208: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547645.824221271: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1734547645.824228150: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1734547645.824236245: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1734547645.824242933: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1734547645.824249364: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1734547645.824257395: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1734547645.824265874: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1734547645.824275441: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1734547645.824283664: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1734547645.824292655: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/snapd/ns
1734547645.824338345: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/bare/5
1734547645.824350407: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1666
1734547645.824359846: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/core22/1720
1734547645.824384707: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5271
1734547645.824393186: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/firefox/5360
1734547645.824399777: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gnome-42-2204/178
1734547645.824406497: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/gtk-common-themes/1535
1734547645.824414048: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/21761
1734547645.824421855: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /snap/snapd/23259
1734547645.824428958: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1734547645.824440700: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/bszk
1734547645.824459642: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /media/lab223/6a0b1392-a961-47f9-ae83-1c4702c6aca2
1734547645.824468121: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547645.824474456: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1734547645.824482199: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.824493078: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1734547645.824499701: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main
1734547645.824679135: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1734547645.824927585: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1734547798.097325437: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x00
1734547798.097386295: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 0
1734547798.098490380: src/main/tools/linux-sandbox.cc:243: child exited normally with code 0

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
	LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
	PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
	PWD=/proc/self/cwd \
	TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/core/conversion/var/BUILD:20:11: Compiling core/conversion/var/Var.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/58/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/core/DeprecatedTypePropertiesRegistry.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:59:11: Compiling core/conversion/converters/NodeConverterRegistry.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/57/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/ops/cudnn_batch_norm_ops.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:39:11: Compiling core/conversion/converters/converter_util.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/56/execroot/_main/external/_main~_repo_rules~libtorch/include/ATen/ops/native_dropout_backward_cpu_dispatch.h (???????)
Target //:libtorchtrt failed to build
INFO: Elapsed time: 1000.299s, Critical Path: 574.06s
INFO: 7984 processes: 7938 internal, 46 linux-sandbox.
ERROR: Build did NOT complete successfully

@narendasan
Copy link
Collaborator

Hmm, how did you install bazel and can you share bazel/bazelisk version numbers?

@breknddone
Copy link
Author

I use this to install bazel：

wget -v https://github.com/bazelbuild/bazelisk/releases/download/v1.20.0/bazelisk-linux-arm64
sudo mv bazelisk-linux-arm64 /usr/bin/bazel
chmod +x /usr/bin/bazel

Hmm, how did you install bazel and can you share bazel/bazelisk version numbers?

@narendasan
Copy link
Collaborator

Can you quickly try bazelisk v1.25.0, we saw some issues with earlier bazelisk versions #3328

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@narendasan @breknddone