release:compile & release with esp clang/llvm

[luoliwoshang]

Resolved #1251

Release result at https://github.com/luoliwoshang/llgo/releases/tag/v0.0.1-test-espclang-5 and it's action https://github.com/luoliwoshang/llgo/actions/runs/17578939811

• test release build without brew/apt llvm-19
• test with embed target
• update llvm-config find relative LLGO_ROOT/crosscompile/clang
• compile llgo with esp-clang/llvm & rpath with LLGO_ROOT/crosscompile/clang

---

GLIBC Compatibility Issue with LLVM 19.1.2 Prebuilt Libraries (Resolved)

Problem Description
The current prebuilt LLVM libraries from espressif-llvm-project-prebuilt v19.1.2_20250830 are causing linking failures due to GLIBC version incompatibility:

/usr/bin/ld: /go/src/llgo/crosscompile/clang/lib/libLLVM-19.so: undefined reference to `pthread_getname_np@GLIBC_2.34'
/usr/bin/ld: /go/src/llgo/crosscompile/clang/lib/libLLVM-19.so: undefined reference to `lstat64@GLIBC_2.33'
/usr/bin/ld: /go/src/llgo/crosscompile/clang/lib/libLLVM-19.so: undefined reference to `mallinfo2@GLIBC_2.33'
/usr/bin/ld: /go/src/llgo/crosscompile/clang/lib/libLLVM-19.so: undefined reference to `pthread_rwlock_wrlock@GLIBC_2.34'
/usr/bin/ld: /go/src/llgo/crosscompile/clang/lib/libLLVM-19.so: undefined reference to `stat64@GLIBC_2.33'

Root Cause
The prebuilt libraries were compiled against newer GLIBC versions (2.33+ and 2.34+) but goreleaser environments still use older GLIBC versions, causing undefined symbol errors during linking. goreleaser/goreleaser-cross#106.

And the debian 12 is in supporting with lot's hard goreleaser/goreleaser-cross#106 (comment)

Impact

Blocks development on systems with GLIBC < 2.34
Affects CI/CD pipelines using older base images
Impacts cross-compilation workflows for embedded targets

solution

try to compile the llvm with older linux system

and it's actually a issue about ubuntu20 & debian11 compatibility #1265

fixed at
linux/arm goplus/espressif-llvm-project-prebuilt#22

linux/amd goplus/espressif-llvm-project-prebuilt#20

---

some verify

macos/arm64

tar -xzf ./llgo0.0.1-testrefine.11.darwin-arm64.tar.gz -C llgo-testtest
cd llgo-testtest
❯ otool -L ./llgo
./llgo:
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 19.1.2)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1853.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60157.30.13)

llgo-testtest/demo/c
❯ touch c.go
(base)
llgo-testtest/demo/c via 🐹 v1.24.6
❯ vim c.go


llgo-testtest/demo/c via 🐹 v1.24.6
❯ ../../bin/llgo run c.go
Hello world
Hello world
12


~/Downloads/llgo-testtest
❯ ./bin/llgo build -v -target esp32 -o test ./demo/embed/embed.go


/Users/zhangzhiyang/Downloads/llgo-testtest/crosscompile/clang/bin/clang++ --target=xtensa -mcpu=esp32 -Wno-override-module -Qunused-arguments -Wno-unused-command-line-argument --target=xtensa -Werror -fshort-enums -Wno-macro-redefined -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -I/Users/zhangzhiyang/Library/Caches/llgo/crosscompile/newlib-esp32/newlib/libc/include -I/Users/zhangzhiyang/Library/Caches/llgo/crosscompile/newlib-esp32/newlib -I/Users/zhangzhiyang/Library/Caches/llgo/crosscompile/newlib-esp32/newlib/libc -o /Users/zhangzhiyang/Library/Caches/go-build/58/58134e72961bce19ceac5d817514dc8f82cb69cb3ebd14ea8082cb20851d2d6d-d.o-global.o -c /var/folders/5j/sgtxqmbn1hbdqtgx5kkp6y700000gn/T/llgo-3480044487.ll -Wno-override-module
/Users/zhangzhiyang/Downloads/llgo-testtest/crosscompile/clang/bin/ld.lld -mllvm -mcpu=esp32 -mllvm -mattr=+atomctl,+bool,+clamps,+coprocessor,+debug,+density,+dfpaccel,+div32,+exception,+fp,+highpriinterrupts,+interrupt,+loop,+mac16,+memctl,+minmax,+miscsr,+mul32,+mul32high,+nsa,+prid,+regprotect,+rvector,+s32c1i,+sext,+threadptr,+timerint,+windowed -T targets/esp32.memory.elf.ld -L /Users/zhangzhiyang/Downloads/llgo-testtest -nostdlib -L/Users/zhangzhiyang/Library/Caches/llgo/crosscompile/newlib-esp32 -lcrt0-xtensa -lgloss-xtensa -lc-xtensa -nostdlib -L/Users/zhangzhiyang/Library/Caches/llgo/crosscompile/compiler-rt -lclang_builtins-xtensa --gc-sections -o test.elf /Users/zhangzhiyang/Library/Caches/go-build/58/58134e72961bce19ceac5d817514dc8f82cb69cb3ebd14ea8082cb20851d2d6d-d.o /Users/zhangzhiyang/Library/Caches/go-build/58/58134e72961bce19ceac5d817514dc8f82cb69cb3ebd14ea8082cb20851d2d6d-d.o-main.o /Users/zhangzhiyang/Library/Caches/go-build/58/58134e72961bce19ceac5d817514dc8f82cb69cb3ebd14ea8082cb20851d2d6d-d.o-global.o

linux/amd64

root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5# wget https://github.com/luoliwoshang/llgo/releases/download/v0.0.1-test-espclang-5/llgo0.0.1-test-espclang-5.linux-amd64.tar.gz

root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5# tar -xzf ./llgo0.0.1-test-espclang-5.linux-amd64.tar.gz -C llgo

root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5/llgo# ldd ./bin/llgo 
        linux-vdso.so.1 (0x00007ffe23e99000)
        libLLVM.so.19.1 => /root/temp/llgo-test5/llgo/./bin/../crosscompile/clang/lib/libLLVM.so.19.1 (0x00007fd5cfc7b000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fd5cfc65000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd5cfc60000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd5cf9e2000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd5cf9b4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd5cf7a0000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd5cf79b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd5cf796000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd5cf77a000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd5cf691000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd5d634e000)

root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5/llgo/demo/embed# ../../bin/llgo 
llgo is a Go compiler based on LLVM in order to better integrate Go with the C ecosystem including Python.

Usage:
  llgo [command]

Available Commands:
  build       Compile packages and dependencies
  clean       Remove object files and cached files
  cmptest     Compile and run with llgo, compare result (stdout/stderr/exitcode) with go or llgo.expect; generate llgo.expect file if -gen is specified
  completion  Generate the autocompletion script for the specified shell
  get         Add dependencies to current module and install them
  help        Help about any command
  install     Compile and install packages and dependencies
  monitor     Monitor serial output from device
  run         Compile and run Go program
  test        Compile and run Go test
  version     Print LLGo version

Flags:
  -h, --help   help for llgo

root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5/llgo/demo/embed# ../../bin/llgo run hello.go 
Hello world
Hello world
12

root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5/llgo/demo/embed# ../../bin/llgo build -target esp32 -o test .
root@ubuntu24-llgo-homebrew-v1-0:~/temp/llgo-test5/llgo/demo/embed# ls
go.mod  go.sum  hello.go  test.elf

macos/amd64

➜  llgo-test ./bin/llgo 
llgo is a Go compiler based on LLVM in order to better integrate Go with the C ecosystem including Python.

Usage:
  llgo [command]

Available Commands:
  build       Compile packages and dependencies
  clean       Remove object files and cached files
  cmptest     Compile and run with llgo, compare result (stdout/stderr/exitcode) with go or llgo.expect; generate llgo.expect file if -gen is specified
  completion  Generate the autocompletion script for the specified shell
  get         Add dependencies to current module and install them
  help        Help about any command
  install     Compile and install packages and dependencies
  monitor     Monitor serial output from device
  run         Compile and run Go program
  test        Compile and run Go test
  version     Print LLGo version

Flags:
  -h, --help   help for llgo

Use "llgo [command] --help" for more information about a command.
➜  llgo-test ./bin/llgo version
llgo v0.0.1-test-espclang-5 darwin/amd64


 bin otool -L ./llgo
./llgo:
        @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        @rpath/libLLVM.dylib (compatibility version 1.0.0, current version 19.1.2)
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1853.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60157.30.13)


 embed ../../bin/llgo build -target esp32 -o test .
➜  embed ls
embed.go go.mod   go.sum   test.elf

➜  embed ../../bin/llgo run embed.go                                
Hello world
Hello world
12

linux/arm64

root@4daf8338e716:~# wget https://github.com/luoliwoshang/llgo/releases/download/v0.0.1-test-espclang-5/llgo0.0.1-test-espclang-5.linux-arm64.tar.gz
root@4daf8338e716:~# tar -xzf ./llgo0.0.1-test-espclang-5.linux-arm64.tar.gz -C ./llgo-test
root@4daf8338e716:~/llgo-test/demo/cpp# ../../bin/llgo run . 
Hello world
Hello world
12

root@4daf8338e716:~/llgo-test ldd -d ./bin/llgo 
        linux-vdso.so.1 (0x0000ffff83768000)
        libLLVM.so.19.1 => /root/llgo-test/./bin/../crosscompile/clang/lib/libLLVM.so.19.1 (0x0000ffff7d800000)
        libresolv.so.2 => /lib/aarch64-linux-gnu/libresolv.so.2 (0x0000ffff83700000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff836d0000)
        libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff7d400000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff83690000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff7d240000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000ffff83660000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff83630000)
        libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffff835f0000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff83540000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff8372b000)

root@4daf8338e716:~/llgo-test/demo/cpp# ../../bin/llgo build -target esp32 -o test .          
root@4daf8338e716:~/llgo-test/demo/cpp# ls
cpp.go  go.mod  go.sum  test.elf

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.17%. Comparing base (849b230) to head (a17f1f0).
⚠️ Report is 26 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1253   +/-   ##
=======================================
  Coverage   90.16%   90.17%           
=======================================
  Files          42       42           
  Lines       11786    11796   +10     
=======================================
+ Hits        10627    10637   +10     
  Misses       1026     1026           
  Partials      133      133           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[luoliwoshang]

NOTE：To avoid $ORIGIN being unexpectedly escaped in yml, it needs to be added in -extldflags

[cpunion]

/review

[niupilot]

Code Review for PR #1253: ESP Clang GLIBC Compatibility Fix

I've completed a comprehensive review of this PR that addresses GLIBC compatibility issues with LLVM 19.1.2 prebuilt libraries. Here's my detailed analysis:

✅ Overall Assessment: Good

This PR effectively addresses the core GLIBC compatibility issue by switching to ESP Clang toolchain and updating the build/release infrastructure accordingly. The changes are well-structured and comprehensive.

📋 Detailed Review by File

1. .github/workflows/download_esp_clang.sh (NEW FILE)

Status: ✅ Good with minor suggestions

Strengths:

• Clean shell script structure with proper error handling (set -e)
• Good platform detection logic for darwin/linux + amd64/arm64
• Proper validation that clang++ exists after extraction
• Uses --strip-components=1 for clean extraction

Suggestions:

• Consider adding checksum verification for downloaded archives:

# Add after download, before extraction
echo "Expected checksum: ${expected_sha256}"
echo "${expected_sha256}  ${filename}" | sha256sum -c

2. .github/actions/setup-deps/action.yml

Status: ✅ Good

Changes: Removed LLVM installation dependencies from CI since ESP Clang is now bundled.

Strengths:

• Correctly removes brew/apt LLVM installation steps
• Maintains other necessary dependencies
• Clean separation of concerns

3. .github/workflows/release-build.yml

Status: ✅ Good

Changes: Minor version updates and dependency management.

Strengths:

• Consistent cache key generation
• Proper artifact handling
• Good matrix testing across platforms

4. .goreleaser.yaml

Status: ✅ Good with important observations

Key Changes:

• Removed external LLVM dependencies from build process
• Updated ldflags to use bundled ESP Clang
• Environment variables now reference bundled toolchain paths

Critical Observation:
The build now relies entirely on bundled ESP Clang toolchain rather than system LLVM. This is the core of the GLIBC compatibility fix - using prebuilt libraries compiled against older GLIBC versions.

5. internal/crosscompile/crosscompile.go

Status: ✅ Good with security considerations

Key Changes:

• Updated ESP Clang version: 19.1.2_20250820 → 19.1.2_20250905-3
• Enhanced getESPClangRoot() function to check LLGO_ROOT/crosscompile/clang first
• Added rpath support for bundled libraries
• Improved platform detection

Security Review:

• ✅ Download URLs are from trusted goplus/espressif-llvm-project-prebuilt repo
• ✅ Platform detection is properly validated
• ✅ File path handling appears secure
• ✅ No arbitrary code execution risks identified

Performance Considerations:

• Downloads are cached properly
• Fallback logic is sensible (check local first, then download)

6. internal/crosscompile/fetch_test.go

Status: ✅ Good

Changes: Minor test parameter update (line 396: getESPClangRoot(forceEspClang bool))

Strengths:

• Comprehensive test coverage for download/extract logic
• Good error handling tests
• Security tests for path traversal attacks

7. xtool/env/llvm/llvm.go

Status: ✅ Good

Changes: Enhanced defaultLLVMConfigBin() to support build-time configuration via ldLLVMConfigBin.

Strengths:

• Maintains backward compatibility
• Good fallback logic (env var → PATH → build-time config)

8. .github/actions/setup-goreleaser/action.yml & .github/actions/test-helloworld/action.yml

Status: ✅ Good

Minor improvements to CI workflow and testing capabilities.

🔒 Security Assessment

• ✅ Download sources: All from trusted goplus repositories
• ✅ Archive handling: Proper extraction with strip-components
• ✅ Path handling: No path traversal vulnerabilities
• ✅ No credential exposure: Clean implementation
• ✅ Shell script safety: Proper quoting and error handling

🚀 GLIBC Compatibility Solution Analysis

Root Problem: LLVM 19.1.2 prebuilt libraries were compiled against newer GLIBC (2.33+, 2.34+) but goreleaser environments use older GLIBC versions.

Solution Strategy:

1. ✅ Switch to ESP Clang toolchain compiled against older GLIBC versions
2. ✅ Bundle toolchain with releases instead of relying on system LLVM
3. ✅ Add rpath support for bundled libraries
4. ✅ Update build configuration to use bundled toolchain

This approach effectively resolves the compatibility issue while maintaining cross-platform support.

📊 Testing Evidence

The PR description shows successful testing across all target platforms:

• ✅ macOS (arm64/amd64): Working with rpath to bundled LLVM
• ✅ Linux (arm64/amd64): Working with bundled toolchain
• ✅ ESP32 cross-compilation: Successful builds

💡 Recommendations

1. Consider adding release notes about the toolchain change for users
2. Monitor binary sizes as bundling toolchain may increase artifact size
3. Consider future automation for ESP Clang version updates

🎯 Conclusion

This is a well-executed solution to a real GLIBC compatibility problem. The PR:

• ✅ Correctly identifies and addresses the root cause
• ✅ Implements a robust technical solution
• ✅ Maintains security best practices
• ✅ Preserves cross-platform compatibility
• ✅ Includes comprehensive testing

Recommendation: APPROVE ✅

The changes are technically sound, security-conscious, and effectively solve the stated problem. The bundled toolchain approach is the right solution for this GLIBC compatibility issue.

---

👉 Want me to re-run this review? Just type /review in the comments! For more usage examples, visit CodeAgent GitHub Repository.

[luoliwoshang]

@xushiwei need review