Building Cgo code depending on shared libraries fails for --remote_download_minimal

[glukasiknuro]

Note that --remote_download_minimal expands to --nobuild_runfile_links, and this option itself triggers the issue.
It was working properly on 0.24.14 version, but is broken in current master.

What version of rules_go are you using?

Current master, testing on rules_go repository.

What version of Bazel are you using?

4.0.0, 5.0.0-pre.20210623.2

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux

What did you do?

Execute rules_go test while using --nobuild_runfile_links

$ git clone https://github.com/bazelbuild/rules_go.git
$ cd rules_go

$ tests/core/cgo/generate_imported_dylib.sh
$ bazel clean && bazel test --nobuild_runfile_links

INFO: From Testing //tests/core/cgo:dylib_test:
==================== Test output for //tests/core/cgo:dylib_test:
(...)/sandbox/linux-sandbox/13/execroot/io_bazel_rules_go/bazel-out/k8-fastbuild/bin/tests/core/cgo/dylib_test_/dylib_test.runfiles/io_bazel_rules_go/tests/core/cgo/dylib_test_/dylib_test: error while loading shared libraries: libimported.so: cannot open shared object file: No such file or directory
================================================================================

Possible regression

Looks like reverting those two lines in https://github.com/bazelbuild/rules_go/pull/2445/files#diff-1fae1aeb9bbd89e02d89a880ef0d1122bd51b265bc281978f01c14fc27f41f47R191 fixes the issue. Based on the comment that was to fix some other issues, but possibly the fix is not entirely valid: #2445 (comment)

Current master:

$ bazel build  //tests/core/cgo:dylib_test
$ readelf -d bazel-bin/tests/core/cgo/dylib_test_/dylib_test
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../../tests/core/cgo:$ORIGIN/dylib_test.runfiles/io_bazel_rules_go/tests/core/cgo]

With revert:

 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../../_solib_k8/_U_S_Stests_Score_Scgo_Clinux_Uimported_Udylib___Utests_Score_Scgo:$ORIGIN/dylib_test.runfiles/io_bazel_rules_go/_solib_k8/_U_S_Stests_Score_Scgo_Clinux_Uimported_Udylib___Utests_Score_Scgo]

Note that after revert the rpath is using mangled names, which seems to be in line with what bazel does for shared libraries: https://github.com/bazelbuild/bazel/blob/c9ab52a294ee490bec43f9ea6e5cf545d97e2ae7/src/main/java/com/google/devtools/build/lib/rules/cpp/SolibSymlinkAction.java#L49.

CC @steeve @jayconrod for any insights why the change was necessary

Other references

• Fix cgo rpath computation for prebuilt dynamic libraries #2869 - There were some issues before, but seems not entirely fixed. CC: @TimothyGu

[glukasiknuro]

Looks like the option --nobuild_runfile_links which the --remote_download_minimal expands to causes the issue.

The issue seem to be somewhat complicated, and the patch in #2907 solves this issue in slightly indirect way. Still it maybe good to apply this patch, to be consistent with how libraries are linked in case of C++.

Description of the current issue

• Current version adds to the created binary RUNPATH two entries, one in runfiles and one outside of it, like: $ORIGIN/dylib_test.runfiles/io_bazel_rules_go/tests/core/cgo and $ORIGIN/../../../../tests/core/cgo
• The binary itself (dylib_test in reproduction) is a symlink. The symlink leads to binary in execroot - outside of the test sandbox (exec root path can be found by bazel info execution_root)
• Runtime dynamic linker bases $ORIGIN on the resolved path to the binary - so it uses binary from execroot, which is outside of the sandbox
• The option --nobuild_runfile_links prevents runfiles creation in execroot, triggering the issue. Without this option the shared library would be found in created runfiles in execroot, but with this options those are not present. (description of when runfiles are created: when to create a runfiles tree bazel#6627)
• Patch in Revert change to use resolved shared lib links #2907 changes dependency of dylib_test - now instead of depending on source library directly, depends on a mangled link that points to the source library. The mangled link is present in the execroot, so resolution of shared library works, using the non-runfiles RUNPATH.
• There are no issues in RBE (remote execution), because the runfiles are not symlinks to execroot but actual files. So this issue is triggered only for tests that are executed locally.