How to run the Example CompilerGym Service

[Angerla]

❓ Questions and Help

I want to run the Example CompilerGym Service, but I'm not clear how to do it. I download the source of the example, the main working directory is as follows::

|-- CompilerGym
|   |-- BUILD.bazel
|   |-- WORKSPACE
|   |-- examples
|   |   |-- example_compiler_gym_service
|   |   |   |-- BUILD
|   |   |   |-- README.md
|   |   |   |-- WORKSPACE
|   |   |   |-- __init__.py
|   |   |   |-- env_tests.py
|   |   |   |-- service_cc
|   |   |   |   |-- BUILD
|   |   |   |   `-- ExampleService.cc
|   |   |   `-- service_py
|   |   |       |-- BUILD
|   |   |       `-- example_service.py
......

I tried this command bazel run -c opt //examples/example_compiler_gym_service/service -- --port=8080 --working_dir=/tmp in the root directory "CompilerGym", but get the error:

ERROR: Skipping '//examples/example_compiler_gym_service/service': no such package 'examples/example_compiler_gym_service/service': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /home/zc/rpc_test/CompilerGym/examples/example_compiler_gym_service/service
WARNING: Target pattern parsing failed.
ERROR: no such package 'examples/example_compiler_gym_service/service': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /home/zc/rpc_test/CompilerGym/examples/example_compiler_gym_service/service
INFO: Elapsed time: 1.316s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

I also tried the command bazel run -c opt //examples/example_compiler_gym_service/service_cc -- --port=8080 --working_dir=/tmp at the root directory and get the error:

ERROR: Skipping '//examples/example_compiler_gym_service/service_cc': no such target '//examples/example_compiler_gym_service/service_cc:service_cc': target 'service_cc' not declared in package 'examples/example_compiler_gym_service/service_cc' defined by /home/zc/rpc_test/CompilerGym/examples/example_compiler_gym_service/service_cc/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target '//examples/example_compiler_gym_service/service_cc:service_cc': target 'service_cc' not declared in package 'examples/example_compiler_gym_service/service_cc' defined by /home/zc/rpc_test/CompilerGym/examples/example_compiler_gym_service/service_cc/BUILD
INFO: Elapsed time: 1.330s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

And I tried the bazel run -c opt //compiler_gym/bin:manual_env -- --service=localhost:8080 at the root directory and meet compiler error:

ERROR: /home/zc/rpc_test/CompilerGym/compiler_gym/envs/llvm/service/BUILD:124:11: Compiling compiler_gym/envs/llvm/service/Cost.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 318 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 318 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from ./compiler_gym/envs/llvm/service/Cost.h:9:0,
                 from compiler_gym/envs/llvm/service/Cost.cc:5:
external/magic_enum/include/magic_enum.hpp: In function 'constexpr auto magic_enum::detail::values(std::index_sequence<I ...>)':
external/magic_enum/include/magic_enum.hpp:419:22: error: the value of 'count' is not usable in a constant expression
   std::array<E, count> values{};
                      ^
external/magic_enum/include/magic_enum.hpp:409:25: note: 'count' used in its own initializer
   constexpr std::size_t count = [](decltype((valid)) valid_) constexpr noexcept -> std::size_t {
                         ^~~~~
external/magic_enum/include/magic_enum.hpp:419:22: note: in template argument for type 'long unsigned int' 
   std::array<E, count> values{};
                      ^
external/magic_enum/include/magic_enum.hpp:422:17: error: invalid types 'int[std::size_t {aka long unsigned int}]' for array subscript
       values[v++] = value<E, Min, IsFlags>(i);
                 ^
'
Internal compiler error: Error reporting routines re-entered.
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Target //compiler_gym/bin:manual_env failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.418s, Critical Path: 2.50s
INFO: 11 processes: 11 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

I'm not clearly how to run the example and hope your help. Thanks.

[ChrisCummins]

Hi @Angerla, thanks for the questions! I'll answer them in reverse order since the build error needs fixing first:

Build error "invalid types ..."

This is likely because the version of GCC you are using is too old to build CompilerGym. CompilerGym needs a GCC >= 9. You can check your version using:

/usr/bin/gcc --version

On Linux we recommend building using clang >= 6. You can force bazel to use a different compiler by setting export CC=/path/to/clang and export CXX=/path/to/clang++. See the INSTALL.md document for further details.

How to run the example service

The example's README is out of date, sorry. The command to run the example C++ service is:

$ bazel run -c opt //examples/example_compiler_gym_service/service_cc:compiler_gym-example-service-cc -- --port=8080 -alsologtostderr -v=3

This will start a service that you can then connect to from python:

>>> from compiler_gym.envs import CompilerEnv
>>> env = CompilerEnv(service="localhost:8080")

However, on its own that probably isn't too useful as you also want to register a dataset and a reward signal as done here.

To use those environments you run your python script using bazel. For example, the unit tests:

$ bazel run -c opt //examples/example_compiler_gym_service:env_tests

You could replace the contents of examples/example/compiler_gym_service/env_tests.py with something like this:

import compiler_gym
import logging
import examples.example_compiler_gym_service  # noqa Register environments

logging.basicConfig(level=logging.DEBUG)

with compiler_gym.make("example-cc-v0") as env:
  env.reset()
  env.step(env.action_space.sample())

Hope that helps!

Cheers,
Chris

[Angerla]

I have changed the version of GCC and run the example C++ service but meet this compiler error:

(compiler_gym) root@0be5e588a6c8:CompilerGym# bazel run -c opt //examples/example_compiler_gym_service/service_cc:compiler_gym-example-service-cc -- --port=8080 -alsologtostderr -v=3
INFO: Analyzed target //examples/example_compiler_gym_service/service_cc:compiler_gym-example-service-cc (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /root/.cache/bazel/_bazel_root/ff8c15b3a8a6d219037f84cf0e5c5d93/external/com_google_absl/absl/hash/BUILD.bazel:98:11: Compiling absl/hash/internal/city.cc [for host] failed: undeclared inclusion(s) in rule '@com_google_absl//absl/hash:city':
this rule is missing dependency declarations for the following files included by 'absl/hash/internal/city.cc':
  '/usr/local/gcc-11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/stdint.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/stdlib.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/cstdlib'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/x86_64-pc-linux-gnu/bits/c++config.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/x86_64-pc-linux-gnu/bits/os_defines.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/x86_64-pc-linux-gnu/bits/cpu_defines.h'
  '/usr/local/gcc-11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include/stddef.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/std_abs.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/utility'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_relops.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_pair.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/move.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/type_traits'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/initializer_list'
  '/usr/local/gcc-11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include-fixed/limits.h'
  '/usr/local/gcc-11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0/include-fixed/syslimits.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/cstddef'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/ciso646'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/algorithm'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_algobase.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/functexcept.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/exception_defines.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/cpp_type_traits.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/ext/type_traits.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/ext/numeric_traits.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_iterator_base_types.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_iterator_base_funcs.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/concept_check.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/debug/assertions.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_iterator.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/ptr_traits.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/debug/debug.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/predefined_ops.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_algo.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/algorithmfwd.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_heap.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_tempbuf.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/stl_construct.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/new'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/exception.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/bits/uniform_int_dist.h'
  '/usr/local/gcc-11.2.0/include/c++/11.2.0/cstdint'
Target //examples/example_compiler_gym_service/service_cc:compiler_gym-example-service-cc failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.873s, Critical Path: 0.32s
INFO: 37 processes: 29 internal, 8 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

The version of GCC is 11.2, the version of Clang is 10, and the version of Bazel is 4.0.

[ChrisCummins]

You will need to clear the bazel cache whenever switching compiler.

bazel clean --expunge

Then repeat your bazel run -c opt //examples/example_compiler_gym_service/service_cc:compiler_gym-example-service-cc -- --port=8080 -alsologtostderr -v=3 command.

Cheers,
Chris

[Angerla]

I meet this compiler problem after I clean the bazel cache:

ERROR: /root/.cache/bazel/_bazel_root/ff8c15b3a8a6d219037f84cf0e5c5d93/external/boringssl/BUILD:131:11: Compiling src/crypto/curve25519/curve25519.c failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 39 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 39 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
external/boringssl/src/crypto/curve25519/curve25519.c:503:57: error: argument 2 of type 'const uint8_t[32]' {aka 'const unsigned char[32]'} with mismatched bound [-Werror=array-parameter=]
  503 | int x25519_ge_frombytes_vartime(ge_p3 *h, const uint8_t s[32]) {
      |                                           ~~~~~~~~~~~~~~^~~~~
In file included from external/boringssl/src/crypto/curve25519/curve25519.c:33:
external/boringssl/src/crypto/curve25519/internal.h:109:58: note: previously declared as 'const uint8_t *' {aka 'const unsigned char *'}
  109 | int x25519_ge_frombytes_vartime(ge_p3 *h, const uint8_t *s);
      |                                           ~~~~~~~~~~~~~~~^
external/boringssl/src/crypto/curve25519/curve25519.c:823:57: error: argument 2 of type 'const uint8_t *' {aka 'const unsigned char *'} declared as a pointer [-Werror=array-parameter=]
  823 | void x25519_ge_scalarmult_base(ge_p3 *h, const uint8_t *a) {
      |                                          ~~~~~~~~~~~~~~~^
In file included from external/boringssl/src/crypto/curve25519/curve25519.c:33:
external/boringssl/src/crypto/curve25519/internal.h:117:56: note: previously declared as an array 'const uint8_t[32]' {aka 'const unsigned char[32]'}
  117 | void x25519_ge_scalarmult_base(ge_p3 *h, const uint8_t a[32]);
      |                                          ~~~~~~~~~~~~~~^~~~~
cc1: all warnings being treated as errors
Target //examples/example_compiler_gym_service/service_cc:compiler_gym-example-service-cc failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 44.013s, Critical Path: 5.07s
INFO: 462 processes: 44 internal, 418 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Thanks.

[ChrisCummins]

I'm not sure about this. It still looks like a problem with the version of gcc you are using. Can you report the output of:

/usr/bin/gcc --version

Cheers,
Chris

[ChrisCummins]

BTW you may find it easier to use the our docker linux image as a build environment. This is the image that we use to build the Linux binary releases. From the root of the repository, run:

# drop into the build environment:
make bdist_wheel-linux-shell

# install the build dependencies in the docker image:
bash packaging/compiler_gym-manylinux-build/container_init.sh

that will drop you into the bash prompt of a linux enviornment with all of the required build dependencies. You can build binaries in the /CompilerGym directory of the container which is shared with the host system.

Cheers,
Chris

[Angerla]

The version of GCC is 11.2:

(base) root@0be5e588a6c8:/# /usr/bin/gcc --version
gcc (GCC) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks, I may try the docker linux image.

[ChrisCummins]

Thanks for reporting back your GCC version. I haven't tested that specific version but it should be fine! Given that the build error is in compiling one of the third party dependencies (the boringssl package), I'm not sure how to fix the problem. The only thing I would suggest would be to use the docker image, or switch to a clang toolchain and start a clean build to see if that addresses the problem.

Cheers,
Chris

[ChrisCummins]

BTW I have updated the documentation for the example service so I'm closing this issue, but feel free to continue commenting here while we resolve your build problems.

Cheers,
Chris