wasmd does not run on centos 7

[westaking]

wasmd does not run on centos7.
This is because of the following reasons.

wasmd: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /home/user777/go/pkg/mod/github.com/confio/[email protected]/api/libgo_cosmwasm.so)

go-cosmwasm builds well on centos7.

libgo_cosmwasm.so file should support centos7.

[westaking]

this is the result of ldd libgo_cosmwasm.so built on centos 7

    linux-vdso.so.1 =>  (0x00007ffc44573000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fb8206db000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fb8204d3000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb8202b7000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb8200a1000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb81fcd3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb820ec3000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fb81f9d1000)

[ethanfrey]

Here is the default build from ubuntu 18.04 (also works on debian):

        linux-vdso.so.1 (0x00007ffe022d2000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff15019c000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff14ff94000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff14fd75000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff14fb5d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff14f76c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff15074c000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff14f3ce000)

[ethanfrey]

The versions all seem to be the same, the issue is the location of the links.
/lib64/xxx.so.1 vs /lib/x86_64-linux-gnu/xxx.so.1

@westaking can you try the following on the master branch (with default build) and let me know if it works for you?

sudo ln -s /lib64 /lib/x86_64-linux-gnu (only works if /lib/x86_64-linux-gnu doesn't exist on your system)

Btw, this is the contents of lib64 on my system:

$ ls -l /lib64
total 0
lrwxrwxrwx 1 root root 32 dic 30  2018 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.27.so

[ethanfrey]

Alternatively, I can try using a static lib on rust. It says how to do so here.. let's see if this works (and doesn't break macos builds):

https://doc.rust-lang.org/1.19.0/reference/linkage.html

[webmaster128]

Linking glibc on Linux is backward-compatible but not forward-compatible. i.e. whatever you build on a given system will run on newer systems but not on older ones. (see also https://stackoverflow.com/a/11108336/2013738)

When the lib is build on Ubuntu 18.04 it will require glibc >= 2.27. But Centos 7 ships glibc 2.12.

In general you want to have a build system that is older or equal to all supported runtime systems. The lowest version that can reasonably be used for building is Ubuntu 16.04 LTS, which sets the min glibc version to 2.23. Using 16.04 would add compatibility to Debian 9 Strech (oldstable).

Given this, I do not have much hope for Centos 7 support using precompiled binaries. But Centos 8 should work (coming with glibc 2.28).

I wonder if statically linking of the libc is even possible. I never heard someone trying that.

---

This behaves entirely different for Windows and Mac where you tell the compiler which min OS version should be supported. At Kullo we compile Windoes 7+ software on Windows 10 and Mac 10.12+ software on Mac 10.15.

[ethanfrey]

Okay, a bit of a trick to get the static lib to work, but maybe that is the proper approach for this. I consider it experimental in any case, but let's see if this allows better cross-linux support.

@westaking Please checkout centos-support branch and try make test without building a local copy

(Note that the libgo_cosmwasm.a file is 27M compared to 3M for libgo_cosmwasm.so... this is not ready to ship but mostly a proof of concept to see what we can do with rust)

[westaking]

The versions all seem to be the same, the issue is the location of the links.
/lib64/xxx.so.1 vs /lib/x86_64-linux-gnu/xxx.so.1

@westaking can you try the following on the master branch (with default build) and let me know if it works for you?

sudo ln -s /lib64 /lib/x86_64-linux-gnu (only works if /lib/x86_64-linux-gnu doesn't exist on your system)

Btw, this is the contents of lib64 on my system:

$ ls -l /lib64
total 0
lrwxrwxrwx 1 root root 32 dic 30  2018 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.27.so

It doesn't work.

[westaking]

Linking glibc on Linux is backward-compatible but not forward-compatible. i.e. whatever you build on a given system will run on newer systems but not on older ones. (see also https://stackoverflow.com/a/11108336/2013738)

I know that..... It doesn't work...

When the lib is build on Ubuntu 18.04 it will require glibc >= 2.27. But Centos 7 ships glibc 2.12.

In general you want to have a build system that is older or equal to all supported runtime systems. The lowest version that can reasonably be used for building is Ubuntu 16.04 LTS, which sets the min glibc version to 2.23. Using 16.04 would add compatibility to Debian 9 Strech (oldstable).

Given this, I do not have much hope for Centos 7 support using precompiled binaries. But Centos 8 should work (coming with glibc 2.28).

I wonder if statically linking of the libc is even possible. I never heard someone trying that.

This behaves entirely different for Windows and Mac where you tell the compiler which min OS version should be supported. At Kullo we compile Windoes 7+ software on Windows 10 and Mac 10.12+ software on Mac 10.15.

Linking glibc on Linux is backward-compatible but not forward-compatible. i.e. whatever you build on a given system will run on newer systems but not on older ones. (see also https://stackoverflow.com/a/11108336/2013738)

[westaking]

Okay, a bit of a trick to get the static lib to work, but maybe that is the proper approach for this. I consider it experimental in any case, but let's see if this allows better cross-linux support.

@westaking Please checkout centos-support branch and try make test without building a local copy

(Note that the libgo_cosmwasm.a file is 27M compared to 3M for libgo_cosmwasm.so... this is not ready to ship but mostly a proof of concept to see what we can do with rust)

It works fine as follows:

1. Build go-cosmwasm in centos7 to create libgo_cosmwasm.so
2. Copy the created libgo_cosmwasm.so to ~/go/pkg/mod/github.com/confio/[email protected]/api
3. Build wasmd.
4. Run wasmd (it works~ yeah~)

It would suggest follows.

1. Create a branch for centos7 in the go-cosmwasm repository
2. Deploy libgo_cosmwasm.so built on centos7.
3. Refer to the centos7 version of go-cosmwasm in go.mod in the wasmd repository (reference based on platform, I don't know if it is possible)

ps. I'm sorry to poor my english.

[ethanfrey]

I'm sorry, I didn't understand if you tested the version with *.a (not building locally).

As to keeping branches, I can make a branch of go-cosmwasm with custom support
Then make a branch of wasmd that refers to that branch
And somehow keep them up to sync with releases.

But that adds a fair bit of overhead. And makes it hard to follow releases. I would ideally get a build system that just works in one branch. But yeah, it is possible one way or another.

If you want to try this out. You can make a fork of go-cosmwasm and maintain that (you have the centos fork, just pull from master and rebuild the custom *.so file). Then make a fork of wasmd, which is the same as upstream (so easy to pull changes), except you add one line to the go.mod file, with a replace directive.

In fact, if you want to use this in the next week, that is likely the only approach. I will investigate a longer term universal solution, but I don't want to maintain branches for support of platforms I can't even test on. I am happy to assist you in supporting that, or work for a more universal solution.

Example use of replace: https://github.com/cosmos/cosmos-sdk/blob/master/go.mod#L35

[westaking]

I have not tested * .a
I tested * .so built in centos7.

It's sad that you can't have continuous support for centos7.

in go.mod of wasmd
replace github.com/confio/go-cosmwasm => github.com/westaking/go-cosmwasm v0.4.1-c7

it works.

thank you for the information.

[ethanfrey]

The issue is the version of glibc as correctly identified by @webmaster128

If we look at debian glibc support:

• jessie, released 2015 and no longer supported, has glibc 2.19
• stretch, the no-longer current release from 2017, has glibc 2.24

I think it is a good idea to compile with an older architecture to get a older glibc version. But forcing back to 2.12 is quite a stretch.

I will try to build the next release with as old as I can, but given jessie is deprecated, I wonder how much tooling works for it.

[ethanfrey]

@westaking There is an option for official support.

I build the releases with a Dockerfile. This is based on Debian Stretch or Buster, and compiles for linux and osx.

If you want to make a PR with a subdir that contains a Dockerfile that builds based on CentOS (no cross-compile - just the linux version), I would be happy to merge it. If the produced build works well on a modern CI machine, I may use this for the official releases. I am not sure if I lose anything with such an old glibc version, but if it works, I can support it.

Since you have installed the rust toolchain and compiled locally in CentOS, this will be much easier for you to get working than me trying to figure it out. I think I will keep the MacOS and later Windows cross-compile from the current Dockerfile, but could use yours for wider linux support

[westaking]

thank you for caring for centos7 users.

what should I do for docker?
let me know in detail and I will try.

[ethanfrey]

Take a look at the current Dockerfile for a rough idea.

You should start FROM centos:centos7 most likely: https://github.com/confio/go-cosmwasm/blob/master/Dockerfile

Then install rustup (and add nightly) inside the Docker image

After that, you probably just want build_linux.sh from here: https://github.com/confio/go-cosmwasm/blob/master/Dockerfile#L49-L58

Minimum you need so that docker run ... will output a lib_gocosmwasm.so file that runs on your system.

[ethanfrey]

Feel free to use or ignore the build script, whatever is easier for you.
Just see if you can get it to work.

You will want to add some other lines in the Makefile, like these: https://github.com/confio/go-cosmwasm/blob/master/Makefile#L52-L56

docker-image-centos, docker-build-centos

If it works and doesn't break anything else, I can merge it and start a bit of cleanup if needed

[westaking]

I created a dockerfile for centos7.
please check this PR : #35

[westaking]

and I tested it by building wasmd with static library in the centos-support branch. (libgo_cosmwasm.a)

# in go.mod (https://github.com/cosmwasm/wasmd/blob/master/go.mod)
replace github.com/confio/go-cosmwasm => github.com/confio/go-cosmwasm centos-support

works fine in centos7 ( with static lib)

If you keep it as static lib, remove PR #35 (dockerfile for centos7)

[ethanfrey]

Thank you very much. I will review it today.

[ethanfrey]

Thank you, I merged #35

I will integrate this into the release build system and include support for CentOS 7 in the next tagged release

[westaking]

wow~ thank you~

[ethanfrey]

Preparing with CentOS7 support on 0.6 release (tomorrow): #42

Thanks for your help