Fails to build with STRICT_ABI option

[nurupo]

While investigating #1261 I have found that toxcore doesn't compile with a STRICT_ABI cmake option due to test binaries not being able to find internal ABI symbols when linking against the shared toxcore.

sudo docker run --rm -it debian:stretch /bin/bash

apt-get update

apt-get install -y \
    build-essential \
    cmake \
    git \
    libopus-dev \
    libsodium-dev \
    libvpx-dev

cd ~

git clone https://github.com/TokTok/c-toxcore
cd c-toxcore
git checkout v0.2.8
mkdir _build
cd _build
cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DENABLE_SHARED=ON \
    -DENABLE_STATIC=OFF \
    -DBUILD_TOXAV=ON \
    -DBOOTSTRAP_DAEMON=OFF \
    -DSTRICT_ABI=ON
make

Output

-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
SOVERSION: 2.8.0
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") 
-- Supported C compiler features = c_function_prototypes;c_restrict;c_variadic_macros;c_static_assert
-- Supported C++ compiler features = cxx_template_template_parameters;cxx_alias_templates;cxx_alignas;cxx_alignof;cxx_attributes;cxx_auto_type;cxx_constexpr;cxx_decltype;cxx_decltype_incomplete_return_types;cxx_default_function_template_args;cxx_defaulted_functions;cxx_defaulted_move_initializers;cxx_delegating_constructors;cxx_deleted_functions;cxx_enum_forward_declarations;cxx_explicit_conversions;cxx_extended_friend_declarations;cxx_extern_templates;cxx_final;cxx_func_identifier;cxx_generalized_initializers;cxx_inheriting_constructors;cxx_inline_namespaces;cxx_lambdas;cxx_local_type_template_args;cxx_long_long_type;cxx_noexcept;cxx_nonstatic_member_init;cxx_nullptr;cxx_override;cxx_range_for;cxx_raw_string_literals;cxx_reference_qualified_functions;cxx_right_angle_brackets;cxx_rvalue_references;cxx_sizeof_member;cxx_static_assert;cxx_strong_enums;cxx_thread_local;cxx_trailing_return_types;cxx_unicode_literals;cxx_uniform_initialization;cxx_unrestricted_unions;cxx_user_literals;cxx_variadic_macros;cxx_variadic_templates;cxx_aggregate_default_initializers;cxx_attribute_deprecated;cxx_binary_literals;cxx_contextual_conversions;cxx_decltype_auto;cxx_digit_separators;cxx_generic_lambdas;cxx_lambda_init_captures;cxx_relaxed_constexpr;cxx_return_type_deduction;cxx_variable_templates
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Checking for one of the modules 'libsodium'
-- Checking for one of the modules 'opus'
-- Checking for one of the modules 'vpx'
-- Checking for one of the modules 'libconfig'
-- Checking for one of the modules 'msgpack'
-- Checking for one of the modules 'opencv'
-- Checking for one of the modules 'portaudio-2.0'
-- Checking for one of the modules 'sndfile'
-- Looking for explicit_bzero
-- Looking for explicit_bzero - not found
-- Looking for memset_s
-- Looking for memset_s - not found
CMAKE_THREAD_LIBS_INIT: -lpthread
-- Checking for gtest
-- Configuring done
-- Generating done
-- Build files have been written to: /root/c-toxcore/_build
root@8e03dd7af4c7:~/c-toxcore/_build# make 
Scanning dependencies of target toxcore_shared
[  1%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/crypto_core.c.o
[  2%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/crypto_core_mem.c.o
[  2%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/logger.c.o
[  3%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/mono_time.c.o
[  4%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/network.c.o
[  5%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/state.c.o
[  5%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/util.c.o
[  6%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/DHT.c.o
[  7%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/LAN_discovery.c.o
[  8%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/ping.c.o
[  8%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/ping_array.c.o
[  9%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/TCP_client.c.o
[ 10%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/TCP_connection.c.o
[ 11%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/TCP_server.c.o
[ 11%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/list.c.o
[ 12%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/net_crypto.c.o
[ 13%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/onion.c.o
[ 14%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/onion_announce.c.o
[ 15%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/onion_client.c.o
[ 15%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/friend_connection.c.o
[ 16%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/friend_requests.c.o
[ 17%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/Messenger.c.o
[ 18%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/group.c.o
[ 18%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/tox_api.c.o
[ 19%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/tox.c.o
[ 20%] Building C object CMakeFiles/toxcore_shared.dir/toxav/audio.c.o
[ 21%] Building C object CMakeFiles/toxcore_shared.dir/toxav/bwcontroller.c.o
[ 21%] Building C object CMakeFiles/toxcore_shared.dir/toxav/groupav.c.o
[ 22%] Building C object CMakeFiles/toxcore_shared.dir/toxav/msi.c.o
[ 23%] Building C object CMakeFiles/toxcore_shared.dir/toxav/ring_buffer.c.o
[ 24%] Building C object CMakeFiles/toxcore_shared.dir/toxav/rtp.c.o
[ 24%] Building C object CMakeFiles/toxcore_shared.dir/toxav/toxav.c.o
[ 25%] Building C object CMakeFiles/toxcore_shared.dir/toxav/toxav_old.c.o
[ 26%] Building C object CMakeFiles/toxcore_shared.dir/toxav/video.c.o
[ 27%] Building C object CMakeFiles/toxcore_shared.dir/toxencryptsave/toxencryptsave.c.o
[ 28%] Linking C shared library libtoxcore.so
[ 28%] Built target toxcore_shared
Scanning dependencies of target misc_tools
[ 29%] Building C object CMakeFiles/misc_tools.dir/testing/misc_tools.c.o
[ 30%] Linking C static library libmisc_tools.a
[ 30%] Built target misc_tools
Scanning dependencies of target random_testing
[ 31%] Building CXX object CMakeFiles/random_testing.dir/testing/random_testing.cc.o
[ 31%] Linking CXX executable random_testing
[ 31%] Built target random_testing
Scanning dependencies of target auto_friend_connection_test
[ 32%] Building C object CMakeFiles/auto_friend_connection_test.dir/auto_tests/friend_connection_test.c.o
[ 33%] Linking C executable auto_friend_connection_test
CMakeFiles/auto_friend_connection_test.dir/auto_tests/friend_connection_test.c.o: In function `main':
friend_connection_test.c:(.text.startup+0x9b): undefined reference to `current_time_monotonic'
friend_connection_test.c:(.text.startup+0xb1): undefined reference to `mono_time_set_current_time_callback'
collect2: error: ld returned 1 exit status
CMakeFiles/auto_friend_connection_test.dir/build.make:98: recipe for target 'auto_friend_connection_test' failed
make[2]: *** [auto_friend_connection_test] Error 1
CMakeFiles/Makefile2:106: recipe for target 'CMakeFiles/auto_friend_connection_test.dir/all' failed
make[1]: *** [CMakeFiles/auto_friend_connection_test.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

We got a new cmake flag AUTOTEST in master which can be used to disable autotests, but even then the build fails due to regulat test programs still being enabled:

-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
SOVERSION: 2.8.0
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") 
-- Supported C compiler features = c_function_prototypes;c_restrict;c_variadic_macros;c_static_assert
-- Supported C++ compiler features = cxx_template_template_parameters;cxx_alias_templates;cxx_alignas;cxx_alignof;cxx_attributes;cxx_auto_type;cxx_constexpr;cxx_decltype;cxx_decltype_incomplete_return_types;cxx_default_function_template_args;cxx_defaulted_functions;cxx_defaulted_move_initializers;cxx_delegating_constructors;cxx_deleted_functions;cxx_enum_forward_declarations;cxx_explicit_conversions;cxx_extended_friend_declarations;cxx_extern_templates;cxx_final;cxx_func_identifier;cxx_generalized_initializers;cxx_inheriting_constructors;cxx_inline_namespaces;cxx_lambdas;cxx_local_type_template_args;cxx_long_long_type;cxx_noexcept;cxx_nonstatic_member_init;cxx_nullptr;cxx_override;cxx_range_for;cxx_raw_string_literals;cxx_reference_qualified_functions;cxx_right_angle_brackets;cxx_rvalue_references;cxx_sizeof_member;cxx_static_assert;cxx_strong_enums;cxx_thread_local;cxx_trailing_return_types;cxx_unicode_literals;cxx_uniform_initialization;cxx_unrestricted_unions;cxx_user_literals;cxx_variadic_macros;cxx_variadic_templates;cxx_aggregate_default_initializers;cxx_attribute_deprecated;cxx_binary_literals;cxx_contextual_conversions;cxx_decltype_auto;cxx_digit_separators;cxx_generic_lambdas;cxx_lambda_init_captures;cxx_relaxed_constexpr;cxx_return_type_deduction;cxx_variable_templates
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Checking for one of the modules 'libsodium'
-- Checking for one of the modules 'opus'
-- Checking for one of the modules 'vpx'
-- Checking for one of the modules 'libconfig'
-- Checking for one of the modules 'msgpack'
-- Checking for one of the modules 'opencv'
-- Checking for one of the modules 'portaudio-2.0'
-- Checking for one of the modules 'sndfile'
-- Looking for explicit_bzero
-- Looking for explicit_bzero - not found
-- Looking for memset_s
-- Looking for memset_s - not found
CMAKE_THREAD_LIBS_INIT: -lpthread
-- Checking for gtest
-- Configuring done
-- Generating done
-- Build files have been written to: /root/c-toxcore/_build
root@8e03dd7af4c7:~/c-toxcore/_build# make
Scanning dependencies of target toxcore_shared
[  2%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/crypto_core.c.o
[  4%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/crypto_core_mem.c.o
[  6%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/logger.c.o
[  8%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/mono_time.c.o
[ 10%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/network.c.o
[ 12%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/state.c.o
[ 14%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/util.c.o
[ 16%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/DHT.c.o
[ 18%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/LAN_discovery.c.o
[ 20%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/ping.c.o
[ 22%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/ping_array.c.o
[ 24%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/TCP_client.c.o
[ 26%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/TCP_connection.c.o
[ 28%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/TCP_server.c.o
[ 30%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/list.c.o
[ 32%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/net_crypto.c.o
[ 34%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/onion.c.o
[ 36%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/onion_announce.c.o
[ 38%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/onion_client.c.o
[ 40%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/friend_connection.c.o
[ 42%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/friend_requests.c.o
[ 44%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/Messenger.c.o
[ 46%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/group.c.o
[ 48%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/tox_api.c.o
[ 51%] Building C object CMakeFiles/toxcore_shared.dir/toxcore/tox.c.o
[ 53%] Building C object CMakeFiles/toxcore_shared.dir/toxav/audio.c.o
[ 55%] Building C object CMakeFiles/toxcore_shared.dir/toxav/bwcontroller.c.o
[ 57%] Building C object CMakeFiles/toxcore_shared.dir/toxav/groupav.c.o
[ 59%] Building C object CMakeFiles/toxcore_shared.dir/toxav/msi.c.o
[ 61%] Building C object CMakeFiles/toxcore_shared.dir/toxav/ring_buffer.c.o
[ 63%] Building C object CMakeFiles/toxcore_shared.dir/toxav/rtp.c.o
[ 65%] Building C object CMakeFiles/toxcore_shared.dir/toxav/toxav.c.o
[ 67%] Building C object CMakeFiles/toxcore_shared.dir/toxav/toxav_old.c.o
[ 69%] Building C object CMakeFiles/toxcore_shared.dir/toxav/video.c.o
[ 71%] Building C object CMakeFiles/toxcore_shared.dir/toxencryptsave/toxencryptsave.c.o
[ 73%] Linking C shared library libtoxcore.so
[ 73%] Built target toxcore_shared
Scanning dependencies of target misc_tools
[ 75%] Building C object CMakeFiles/misc_tools.dir/testing/misc_tools.c.o
[ 77%] Linking C static library libmisc_tools.a
[ 77%] Built target misc_tools
Scanning dependencies of target save-generator
[ 79%] Building C object CMakeFiles/save-generator.dir/other/fun/save-generator.c.o
[ 81%] Linking C executable save-generator
[ 81%] Built target save-generator
Scanning dependencies of target random_testing
[ 83%] Building CXX object CMakeFiles/random_testing.dir/testing/random_testing.cc.o
[ 85%] Linking CXX executable random_testing
[ 85%] Built target random_testing
Scanning dependencies of target DHT_test
[ 87%] Building C object CMakeFiles/DHT_test.dir/testing/DHT_test.c.o
[ 89%] Linking C executable DHT_test
CMakeFiles/DHT_test.dir/testing/DHT_test.c.o: In function `print_assoc.constprop.0':
DHT_test.c:(.text+0x160): undefined reference to `net_ntohs'
DHT_test.c:(.text+0x172): undefined reference to `ip_ntoa'
DHT_test.c:(.text+0x1b6): undefined reference to `net_ntohs'
DHT_test.c:(.text+0x1cc): undefined reference to `ip_ntoa'
CMakeFiles/DHT_test.dir/testing/DHT_test.c.o: In function `print_assoc.constprop.1':
DHT_test.c:(.text+0x220): undefined reference to `net_ntohs'
DHT_test.c:(.text+0x232): undefined reference to `ip_ntoa'
DHT_test.c:(.text+0x276): undefined reference to `net_ntohs'
DHT_test.c:(.text+0x28c): undefined reference to `ip_ntoa'
CMakeFiles/DHT_test.dir/testing/DHT_test.c.o: In function `main':
DHT_test.c:(.text.startup+0x4b): undefined reference to `ip_init'
DHT_test.c:(.text.startup+0x50): undefined reference to `mono_time_new'
DHT_test.c:(.text.startup+0x74): undefined reference to `new_networking'
DHT_test.c:(.text.startup+0x8a): undefined reference to `new_dht'
DHT_test.c:(.text.startup+0xa3): undefined reference to `dht_get_self_public_key'
DHT_test.c:(.text.startup+0x181): undefined reference to `dht_addfriend'
DHT_test.c:(.text.startup+0x1bd): undefined reference to `net_htons'
DHT_test.c:(.text.startup+0x1e8): undefined reference to `dht_bootstrap_from_address'
DHT_test.c:(.text.startup+0x222): undefined reference to `mono_time_update'
DHT_test.c:(.text.startup+0x22a): undefined reference to `do_dht'
DHT_test.c:(.text.startup+0x232): undefined reference to `dht_get_net'
DHT_test.c:(.text.startup+0x23c): undefined reference to `networking_poll'
DHT_test.c:(.text.startup+0x264): undefined reference to `dht_get_close_client'
DHT_test.c:(.text.startup+0x276): undefined reference to `public_key_cmp'
DHT_test.c:(.text.startup+0x2ec): undefined reference to `dht_get_num_friends'
DHT_test.c:(.text.startup+0x328): undefined reference to `dht_get_friend_public_key'
DHT_test.c:(.text.startup+0x358): undefined reference to `dht_get_friend_public_key'
DHT_test.c:(.text.startup+0x36b): undefined reference to `dht_getfriendip'
DHT_test.c:(.text.startup+0x378): undefined reference to `net_ntohs'
DHT_test.c:(.text.startup+0x38c): undefined reference to `ip_ntoa'
DHT_test.c:(.text.startup+0x3d2): undefined reference to `dht_get_friend'
DHT_test.c:(.text.startup+0x3dd): undefined reference to `dht_friend_client'
DHT_test.c:(.text.startup+0x3ef): undefined reference to `public_key_cmp'
collect2: error: ld returned 1 exit status
CMakeFiles/DHT_test.dir/build.make:98: recipe for target 'DHT_test' failed
make[2]: *** [DHT_test] Error 1
CMakeFiles/Makefile2:144: recipe for target 'CMakeFiles/DHT_test.dir/all' failed
make[1]: *** [CMakeFiles/DHT_test.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

[nurupo]

Labeling it as high since it supposedly hinders packaging of toxcore, as @yangf, who opened the #1261 issue, is the package maintainer of libtoxcore in Debian.

[zugz]

So it looks like the solution is to make building testing/* optional, and have STRICT_ABI imply building neither these nor the autotests?

[nurupo]

I think so, yes.

autotests (the automated test suite) and regular test (the testing programs) are linked against the toxcore and both call into toxcore's internals. When toxcore is built with STRICT_ABI option, none of those internals are exposed, resulting in failed linking and in overall build failure. One way to fix the build failure is not to build autotests and tests when STRICT_ABI is enabled.

Of course there are other ways to work around this rather than disabling building of autotests and tests, like always statically compiling toxcore into the autotests and tests -- this is actually what tox-bootstrapd does. I think disabling building of tests is an acceptable solution, not to mention we wanted to be able to disable tests for a while now, so two birds with one stone two features/bugfixes with one change.