Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[question] Conan + Meson + sysroot issue #16468

Closed
1 task done
donlk opened this issue Jun 12, 2024 · 14 comments
Closed
1 task done

[question] Conan + Meson + sysroot issue #16468

donlk opened this issue Jun 12, 2024 · 14 comments
Assignees

Comments

@donlk
Copy link

donlk commented Jun 12, 2024

What is your question?

Hi!
I'm trying to build Qt with a prebuilt GCC compiler, having an external sysroot containing libc and relevant libraries (pthread, libm, libdl, etc), meaning that the toolchain itself does not have these libraries inside.

So I defined a toolchain profile with a sysroot:

...
[buildenv]
PATH=+(path){{toolchain_path}}/bin
LD_LIBRARY_PATH=+(path){{toolchain_path}}/lib
C_INCLUDE_PATH=+(path){{sysroot}}/usr/include
CC={{toolchain_path}}/bin/{{cc_compiler}}
CXX={{toolchain_path}}/bin/{{cxx_compiler}}
LD={{toolchain_path}}/bin/ld
AR={{toolchain_path}}/bin/ar
AS={{toolchain_path}}/bin/as
NM={{toolchain_path}}/bin/nm
RANLIB={{toolchain_path}}/bin/ranlib
STRIP={{toolchain_path}}/bin/strip
LDFLAGS=-Wl,-rpath-link {{sysroot}}/lib

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux

[conf]
tools.build:sysroot={{sysroot}}
tools.build:sharedlinkflags=["-Wl,-rpath-link {{sysroot}}/lib"]

So far every dependency built successfully, except freetype:

[2/4] Compiling C object libfreetype.a.p/src_bzip2_ftbzip2.c.o
FAILED: libfreetype.a.p/src_bzip2_ftbzip2.c.o
<toolchain_path>/gcc-9.3.0-x86_64/bin/gcc -Ilibfreetype.a.p -I. -I../src -I../src/include -I<sysroot_path>/<home>/conan/home/p/b/zlib337c8139e577f/p/include
...
 -MD -MQ libfreetype.a.p/src_bzip2_ftbzip2.c.o -MF libfreetype.a.p/src_bzip2_ftbzip2.c.o.d -o libfreetype.a.p/src_bzip2_ftbzip2.c.o -c ../src/src/bzip2/ftbzip2.c
../src/src/bzip2/ftbzip2.c:46:10: fatal error: bzlib.h: No such file or directory

The problem is this line (replaced the hard-coded paths with a reference):

-I<sysroot_path>/<home>/conan/home/p/b/zlib337c8139e577f/p/include

The sysroot path is directly prepended to all dependent includes, zlib included, making the zlib header lookup messed up.
I've done some research and found a relevant official meson issue. It says:

Now the issues is, that pkgconf PREFIXES every path in the *.pc files inside dependencies with the sysroot defined in sys_root. However the paths defined pc files in the folder dependencies MUST not be prefixed. Else the #include and linking will fail.

Here's the pc file for zlib:
build-release/conan/zlib.pc:

prefix=<home>/conan/home/p/b/zlib337c8139e577f/p
libdir=${prefix}/lib
includedir=${prefix}/include
bindir=${prefix}/bin

Name: zlib
Description: Conan package: zlib
Version: 1.3.1
Libs: -L"${libdir}" -lz
Cflags: -I"${includedir}"

I've tried nullifying the PKG_CONFIG_SYSROOT_DIR variable via [buildenv] in my profile to force pkgconfig to disregard it, to no avail.
Any ideas? This is a very basic cross-compile issue for conan -> meson, it should be straightforward.

Have you read the CONTRIBUTING guide?

  • I've read the CONTRIBUTING guide
@donlk
Copy link
Author

donlk commented Jun 12, 2024

Another relvevant issue:
pkgconf/pkgconf#213

@franramirez688
Copy link
Contributor

Hi @donlk

Thanks a lot for reporting this! Yes, you're completely right. A way to work around this could be:

  • Add an extra machine file to override the sys_root Meson property:
$ conan [YOUR COMMAND] -c "tools.meson.mesontoolchain:extra_machine_files=['/path/to/file/sys_root_empty.ini']"

Where that sys_root_empty.ini looks like:

[properties]
sys_root = ''

Let me know if it worked or if we need to look into it more.

@donlk
Copy link
Author

donlk commented Jul 4, 2024

Works perfectly. Thank you!

@donlk donlk closed this as completed Jul 4, 2024
@franramirez688
Copy link
Contributor

That's great! 👏

@jwillikers
Copy link
Contributor

jwillikers commented Jul 16, 2024

@franramirez688 While this solution works for Conan packages built with conan create, projects developed using conan install will need to manually add the extra flag to add this ini file as part of the configure step, which is unfortunate.

Both of the linked issues in this thread are relevant, but closed. I think we should try to make sure that Meson and pkgconf are still aware that this is a problem and track open related upstream issues to be sure it gets fixed.

@jwillikers
Copy link
Contributor

For future reference, here's how I removed the sys_root line from the cross file in the generate method.

    def generate(self):
        # Remove the troublesome sys_root parameter from the Meson cross file.
        # It causes pkgconf to prepend the sysroot to the absolute path to the package in the Conan cache.
        if cross_building(self):
            with open(os.path.join(self.generators_folder, "conan_meson_cross.ini"), "r") as f:
                lines = f.readlines()
            with open(os.path.join(self.generators_folder, "conan_meson_cross.ini"), "w") as f:
                for line in lines:
                    if "sys_root = '" not in line:
                        f.write(line)

@donlk
Copy link
Author

donlk commented Jul 16, 2024

projects developed using conan install will need to manually add the extra flag to add this ini file as part of the configure step, which is unfortunate.

That is weird, conan install accepts profile and config attributes as well, in theory it should work, right? Either via -c or in the profile itself.
Either of these failed?

@franramirez688
Copy link
Contributor

@jwillikers I don't get what you mean. So, why is it not working using the conan install? Could you show me an example?

@jwillikers
Copy link
Contributor

jwillikers commented Jul 19, 2024

conan create run's the Meson configure step, where it adds the extra --cross-file parameter automatically, i.e. meson setup build-debug --cross-file build-debug/conan/conan_meson_cross.ini --cross-file ~/.conan2/toolchains/sys_root_empty.ini.

That's all good and dandy, until you are developing a Meson project locally. Developers use conan install followed by the meson commands to configure and build the project. When cross-compiling such a project, developers now have to go from running meson setup build-debug --cross-file build-debug/conan/conan_meson_cross.ini to instead running meson setup build-debug --cross-file build-debug/conan/conan_meson_cross.ini --cross-file ~/.conan2/toolchains/sys_root_empty.ini, which is not convenient.

Also, for reference, I found PR pkgconf/pkgconf#280 that appears to address this issue and is still open at this time.

@wdobbe
Copy link

wdobbe commented Sep 12, 2024

A way to work around this could be:

* Add an extra machine file to override the `sys_root` Meson property:
$ conan [YOUR COMMAND] -c "tools.meson.mesontoolchain:extra_machine_files=['/path/to/file/sys_root_empty.ini']"

Where that sys_root_empty.ini looks like:

[properties]
sys_root = ''

Let me know if it worked or if we need to look into it more.

This doesn't work (anymore) for me. Not even when I add the following line to the recipe of the cross-compiled package (fontconfig in my case):

tc.properties["sys_root"] = ""

With this line the sys_root property in the generated cross-file becomes empty but still the sys-root is prepended to all include paths:

freetype/2.13.2: Meson configure cmd: meson setup --cross-file "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release/conan/conan_meson_cross.ini" --cross-file "/home/wdobbe/.conan2/meson_sys_root_empty.ini" "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release" "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/src" --prefix=/
freetype/2.13.2: RUN: meson setup --cross-file "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release/conan/conan_meson_cross.ini" --cross-file "/home/wdobbe/.conan2/meson_sys_root_empty.ini" "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release" "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/src" --prefix=/
WARNING: Unknown CPU family cortexa9hf-neon, please report this at https://github.com/mesonbuild/meson/issues/new
The Meson build system
Version: 1.3.2
Source dir: /home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/src
Build dir: /home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release
Build type: cross build
Project name: freetype2
Project version: 2.13.2
C compiler for the host machine: arm-poky-linux-gnueabi-gcc (gcc 9.2.0 "arm-poky-linux-gnueabi-gcc (GCC) 9.2.0")
C linker for the host machine: arm-poky-linux-gnueabi-gcc ld.bfd 2.32.0.20190204
C compiler for the build machine: cc (gcc 7.5.0 "cc (SUSE Linux) 7.5.0")
C linker for the build machine: cc ld.bfd 2.41.0.20230908-150100
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: cortexa9hf-neon
Host machine cpu: cortexa9hf-neon
Target machine cpu family: cortexa9hf-neon
Target machine cpu: cortexa9hf-neon
Program python3 found: YES (/usr/local/python39env/bin/python3)
Has header "unistd.h" : YES 
Has header "fcntl.h" : YES 
Has header "sys/mman.h" : YES 
Found pkg-config: YES (/home/wdobbe/.conan2/p/b/pkgco6b2707c42a451/p/bin/pkgconf) 2.1.0
Run-time dependency zlib found: YES 1.2.13
Run-time dependency bzip2 found: YES 1.0.8
Run-time dependency libpng found: YES 1.6.42
Dependency harfbuzz skipped: feature harfbuzz disabled
Run-time dependency libbrotlidec found: YES 1.1.0
Build targets in project: 5

freetype2 2.13.2

  Operating System
    OS         : linux

  Used Libraries
    Zlib       : system
    Bzip2      : yes
    Png        : yes
    Harfbuzz   : no
    Brotli     : yes

  User defined options
    Cross files: /home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release/conan/conan_meson_cross.ini
                 /home/wdobbe/.conan2/meson_sys_root_empty.ini
    prefix     : /

Found ninja-1.12.1 at /home/wdobbe/.conan2/p/b/ninjabf76fb6402bba/p/bin/ninja
                                                                                                                                                                                                                  
freetype/2.13.2: Meson build cmd: meson compile -C "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release" -j64
freetype/2.13.2: RUN: meson compile -C "/home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release" -j64
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/wdobbe/.conan2/p/b/ninjabf76fb6402bba/p/bin/ninja -j 64
[18/46] Compiling C object libfreetype.a.p/src_sfnt_sfnt.c.o
FAILED: libfreetype.a.p/src_sfnt_sfnt.c.o 
arm-poky-linux-gnueabi-gcc -Ilibfreetype.a.p -I. -I../src -I../src/include -I/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi/home/wdobbe/.conan2/p/b/zlib7dbebab8830af/p/include -I/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi/home/wdobbe/.conan2/p/b/bzip2e92281329ac89/p/include -I/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi/home/wdobbe/.conan2/p/b/libpne0671a1ce0e00/p/include -I/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi/home/wdobbe/.conan2/p/b/brotl76641bb1988e1/p/include -I/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi/home/wdobbe/.conan2/p/b/brotl76641bb1988e1/p/include/brotli -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -marm -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong -Wformat -Wformat-security -Werror=format-security --sysroot=/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi -O2 -pipe -feliminate-unused-debug-types -marm -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi -O2 -feliminate-unused-debug-types --sysroot=/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi -fPIC '-DFT_CONFIG_MODULES_H=<ftmodule.h>' -DFT2_BUILD_LIBRARY=1 '-DFT_CONFIG_OPTIONS_H=<ftoption.h>' '-DFT_CONFIG_CONFIG_H=<ftconfig.h>' -MD -MQ libfreetype.a.p/src_sfnt_sfnt.c.o -MF libfreetype.a.p/src_sfnt_sfnt.c.o.d -o libfreetype.a.p/src_sfnt_sfnt.c.o -c ../src/src/sfnt/sfnt.c
In file included from ../src/src/sfnt/sfnt.c:25:
../src/src/sfnt/sfwoff2.c:27:10: fatal error: brotli/decode.h: No such file or directory
   27 | #include <brotli/decode.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
[45/46] Compiling C object libfreetype.a.p/src_truetype_truetype.c.o
ninja: build stopped: subcommand failed.

freetype/2.13.2: ERROR: 
Package 'a801309751333469de9145f28887e6fc7fe500a8' build failed
freetype/2.13.2: WARN: Build folder /home/wdobbe/.conan2/p/b/freet6b9a8b9ee6f3a/b/build-release
ERROR: freetype/2.13.2: Error in build() method, line 114
        meson.build()
        ConanException: Error 1 while executing

@memsharded
Copy link
Member

Hi @wdobbe

Thanks for the feedback.

Could you please clarify if this is a regression, it worked in previous Conan 2.6 and now in 2.7 is broken?
It would be worth to create a new ticket with the details to reproduce

@wdobbe
Copy link

wdobbe commented Sep 13, 2024

Hi @memsharded

While trying to add cross-compile support for the Qt6 CCI recipe I ran into the same problem as reported in this issue: for all dependencies that use meson as build system the sys-root path is prepended to the include path.
So
-I/home/wdobbe/.conan2/p/b/zlib134522c449e8d/p/include
becomes
-I/home/wdobbe/.conan2/p/b/swpt_fb08c05e04578/p/sysroots/cortexa9hf-neon-poky-linux-gnueabi/home/wdobbe/.conan2/p/b/zlib134522c449e8d/p/include

This is not a regression of Conan, it also happens with for example Conan 2.4. From the discussion above it seems to be a pkgconf problem.

What is a kind of 'regression' is that the workaround suggested above (add an extra meson machine file to override the sys_root property) does not work anymore.

So currently I am stuck trying to cross-compile for example freetype that is a dependency for Qt6.
I am now trying to debug Meson to see if I can find another workaround but I am not very familiar with Meson.

I might create a new issue for this later.

@memsharded
Copy link
Member

What is a kind of 'regression' is that the workaround suggested above (add an extra meson machine file to override the sys_root property) does not work anymore.

If you file a new ticket, it would be useful to know in which previous version this was working and when it stopped working. Thanks!

@wdobbe
Copy link

wdobbe commented Sep 16, 2024

If anyone encounters the same problem: cause was that the toolchain I used exported environment variable PKG_CONFIG_SYSROOT_DIR. This causes pkg-config to prepend the sys-root path to any include path.
After removing the PKG_CONFIG_SYSROOT_DIR from the toolchain problem disappeared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants