opam install fails on macos

[benkb]

command:

opam update
opam install luv

error (complete error log below):

+- The following actions failed
| - build luv 0.5.0
- No changes have been performed

specs:

Mac OS X
  ProductVersion:	10.13.6
opam --version 
  2.0.6
ocaml --version
  The OCaml toplevel, version 4.09.0
gcc --version
  Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
  Apple LLVM version 10.0.0 (clang-1000.10.44.4)
  Target: x86_64-apple-darwin17.7.0
  Thread model: posix
  InstalledDir: /Library/Developer/CommandLineTools/usr/bin
python --version
  Python 2.7.16
python3 --version
  Python 3.7.6
which gyp
  /usr/local/bin/gyp

error log:

The compilation of luv failed at
        "/Users/bkb/.opam/opam-init/hooks/sandbox.sh build dune build -p luv -j
 
while compiling luv.0.5.0�
     2.0.6 | macos/x86_64 | ocaml-base-compiler.4.09.0 | https://opam.ocaml.org#bea146b8
mpath�        ~/.opam/default/.opam-switch/build/luv.0.5.0
command�    ~/.opam/opam-init/hooks/sandbox.sh build dune build -p luv -j 7
exit-code�   1
env-file�    ~/.opam/log/luv-83854-1872ec.env
output-file� ~/.opam/log/luv-83854-1872ec.out
 output ###
c_generated_functions.c:1636:36: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
[...]

vendor/libuv/include/uv.h:1205:52: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_os_getenv(const char* name, char* buffer, size_t* size);
                                                   ^
c_generated_functions.c:1669:29: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1276 = luv_os_uname(x1275);

./helpers.h:167:24: note: passing argument to parameter 'buffer' here
int luv_os_uname(char *buffer);
                       ^
15 warnings generated.

[aantron]

I am seeing only warnings here; these may be caused by the recent ctypes release. Can you show which version of ctypes is installed with opam list --installed ctypes?

Do you see any errors in the output-file mentioned in the message, ~/.opam/log/luv-83854-1872ec.out, or a new output file from a new run?

[benkb]

I try now to build things locally with:

git clone https://github.com/aantron/luv.git --recursive
cd luv
opam install --deps-only .

opam is not building anything at this point, is this expected?

opam install --deps-only .
Nothing to do.

but when I try to run an example with dune, there are some more specific errors (see below).

One source of problems could be the line in src/c/dune where the build command for gyp is defined (I am guessing)

...
(run python3 gyp_uv.py -f make -Duv_library=static_library))
...

This script doesnt run, more specific the arguments for make could be wrong

dune exec example/delay.exe

     python3 src/c/dlluv.so,src/c/libuv.a (exit 1)
(cd _build/default/src/c/vendor/libuv && /usr/local/bin/python3 gyp_uv.py -f make -Duv_library=static_library)
['-f', 'make', '-Duv_library=static_library', '-I', 'common.gypi', 'test/test.gyp', '--depth=.', '-Goutput_dir=/Users/bkb/local/libuv/luv/_build/default/src/c/vendor/libuv/out', '--generator-output', '/Users/bkb/local/libuv/luv/_build/default/src/c/vendor/libuv/out', '-Dhost_arch=x64', '-Dtarget_arch=x64']
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
Traceback (most recent call last):
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 1398, in XcodeVersion
    version_list = GetStdout(['xcodebuild', '-version']).splitlines()
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 1454, in GetStdout
    sys.stderr.write(out + b'\n')
TypeError: write() argument must be str, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gyp_uv.py", line 73, in <module>
    run_gyp(gyp_args)
  File "gyp_uv.py", line 38, in run_gyp
    rc = gyp.main(args)
  File "./build/gyp/pylib/gyp/__init__.py", line 545, in main
    return gyp_main(args)
  File "./build/gyp/pylib/gyp/__init__.py", line 530, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "./build/gyp/pylib/gyp/generator/make.py", line 2210, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "./build/gyp/pylib/gyp/generator/make.py", line 820, in Write
    self.Pchify))
  File "./build/gyp/pylib/gyp/generator/make.py", line 1215, in WriteSources
    cflags = self.xcode_settings.GetCflags(configname)
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 630, in GetCflags
    archs = self.GetActiveArchs(self.configname)
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 488, in GetActiveArchs
    xcode_archs_default = GetXcodeArchsDefault()
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 120, in GetXcodeArchsDefault
    xcode_version, _ = XcodeVersion()
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 1409, in XcodeVersion
    version = re.match(r'(\d\.\d\.?\d*)', version).groups()[0]
AttributeError: 'NoneType' object has no attribute 'groups'

[aantron]

opam is not building anything at this point, is this expected?

Yes, because your earlier attempt to install from opam directly already installed the dependencies of luv, and the --deps-only is only trying to install those deps again.

I see this underlying error:

xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory

Do you have Xcode installed? The build requires it at the moment. See #31.

[aantron]

As for this error,

TypeError: write() argument must be str, not bytes

@sikanhe has also observed that. I've tried to reproduce it by artificially having the Xcode check fail in Travis, but haven't been able to do so. It's probably related to the recent Python 3 upgrade upstream in gyp, but, in part because I haven't reproduced it, I can't figure out what is really causing it.

[benkb]

opam list --installed ctypes

ctypes 0.17.1

[aantron]

If Xcode is a pain for you, please let me know. I'll try to adjust the Mac build so Xcode is no longer required (this also slightly depends on messing with upstream).

[aantron]

ctypes 0.17.1

If I understand the ctypes 0.17.0 change correctly, it's designed for no warnings on 4.10.

• with 0.17.0 should build silently on 4.10.
• with < 0.17.0 should build silently on < 4.10.

Either way, these are only warnings, so they are a red herring for the issue. The Xcode message is the true issue :)

[benkb]

Do you see any errors in the output-file mentioned in the message, ~/.opam/log/luv-83854-1872ec.out, or a new output file from a new run?

     python3 src/c/dlluv.so,src/c/libuv.a (exit 1)
(cd _build/default/src/c/vendor/libuv && /usr/local/bin/python3 gyp_uv.py -f make -Duv_library=static_library)
['-f', 'make', '-Duv_library=static_library', '-I', 'common.gypi', 'test/test.gyp', '--depth=.', '-Goutput_dir=/Users/bkb/.opam/default/.opam-switch/build/luv.0.5.0/_build/default/src/c/vendor/libuv/out', '--generator-output', '/Users/bkb/.opam/default/.opam-switch/build/luv.0.5.0/_build/default/src/c/vendor/libuv/out', '-Dhost_arch=x64', '-Dtarget_arch=x64']
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
Traceback (most recent call last):
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 1398, in XcodeVersion
    version_list = GetStdout(['xcodebuild', '-version']).splitlines()
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 1454, in GetStdout
    sys.stderr.write(out + b'\n')
TypeError: write() argument must be str, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gyp_uv.py", line 73, in <module>
    run_gyp(gyp_args)
  File "gyp_uv.py", line 38, in run_gyp
    rc = gyp.main(args)
  File "./build/gyp/pylib/gyp/__init__.py", line 545, in main
    return gyp_main(args)
  File "./build/gyp/pylib/gyp/__init__.py", line 530, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "./build/gyp/pylib/gyp/generator/make.py", line 2210, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "./build/gyp/pylib/gyp/generator/make.py", line 820, in Write
    self.Pchify))
  File "./build/gyp/pylib/gyp/generator/make.py", line 1215, in WriteSources
    cflags = self.xcode_settings.GetCflags(configname)
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 630, in GetCflags
    archs = self.GetActiveArchs(self.configname)
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 488, in GetActiveArchs
    xcode_archs_default = GetXcodeArchsDefault()
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 120, in GetXcodeArchsDefault
    xcode_version, _ = XcodeVersion()
  File "./build/gyp/pylib/gyp/xcode_emulation.py", line 1409, in XcodeVersion
    version = re.match(r'(\d\.\d\.?\d*)', version).groups()[0]
AttributeError: 'NoneType' object has no attribute 'groups'
      ocamlc src/c/c_generated_functions.o
c_generated_functions.c:13:22: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   uv_strerror_r(x4, x7, x8);
                     ^~
vendor/libuv/include/uv.h:388:46: note: passing argument to parameter 'buf' here
UV_EXTERN char* uv_strerror_r(int err, char* buf, size_t buflen);
                                             ^
c_generated_functions.c:21:23: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   uv_err_name_r(x15, x18, x19);
                      ^~~
vendor/libuv/include/uv.h:391:46: note: passing argument to parameter 'buf' here
UV_EXTERN char* uv_err_name_r(int err, char* buf, size_t buflen);
                                             ^
c_generated_functions.c:664:41: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x489 = uv_pipe_getsockname(x486, x487, x488);
                                        ^~~~
vendor/libuv/include/uv.h:762:41: note: passing argument to parameter 'buffer' here
                                  char* buffer,
                                        ^
c_generated_functions.c:672:41: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x496 = uv_pipe_getpeername(x493, x494, x495);
                                        ^~~~
vendor/libuv/include/uv.h:765:41: note: passing argument to parameter 'buffer' here
                                  char* buffer,
                                        ^
c_generated_functions.c:1446:35: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1109 = uv_ip4_name(x1104, x1105, x1106);
                                  ^~~~~
vendor/libuv/include/uv.h:1597:64: note: passing argument to parameter 'dst' here
UV_EXTERN int uv_ip4_name(const struct sockaddr_in* src, char* dst, size_t size);
                                                               ^
c_generated_functions.c:1454:35: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1118 = uv_ip6_name(x1113, x1114, x1115);
                                  ^~~~~
vendor/libuv/include/uv.h:1598:65: note: passing argument to parameter 'dst' here
UV_EXTERN int uv_ip6_name(const struct sockaddr_in6* src, char* dst, size_t size);
                                                                ^
c_generated_functions.c:1567:41: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1203 = uv_if_indextoname(x1198, x1201, x1202);
                                        ^~~~~
vendor/libuv/include/uv.h:1632:39: note: passing argument to parameter 'buffer' here
                                char* buffer,
                                      ^
c_generated_functions.c:1575:40: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1212 = uv_if_indextoiid(x1207, x1210, x1211);
                                       ^~~~~
vendor/libuv/include/uv.h:1635:38: note: passing argument to parameter 'buffer' here
                               char* buffer,
                                     ^
c_generated_functions.c:1582:34: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1217 = uv_os_gethostname(x1215, x1216);
                                 ^~~~~
vendor/libuv/include/uv.h:1220:39: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_os_gethostname(char* buffer, size_t* size);
                                      ^
c_generated_functions.c:1589:27: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1222 = uv_exepath(x1220, x1221);
                          ^~~~~
vendor/libuv/include/uv.h:1638:32: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_exepath(char* buffer, size_t* size);
                               ^
c_generated_functions.c:1596:23: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1227 = uv_cwd(x1225, x1226);
                      ^~~~~
vendor/libuv/include/uv.h:1640:28: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_cwd(char* buffer, size_t* size);
                           ^
c_generated_functions.c:1609:30: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1235 = uv_os_homedir(x1233, x1234);
                             ^~~~~
vendor/libuv/include/uv.h:1173:35: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_os_homedir(char* buffer, size_t* size);
                                  ^
c_generated_functions.c:1616:29: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1240 = uv_os_tmpdir(x1238, x1239);
                            ^~~~~
vendor/libuv/include/uv.h:1174:34: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_os_tmpdir(char* buffer, size_t* size);
                                 ^
c_generated_functions.c:1636:36: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1253 = uv_os_getenv(x1250, x1251, x1252);
                                   ^~~~~
vendor/libuv/include/uv.h:1205:52: note: passing argument to parameter 'buffer' here
UV_EXTERN int uv_os_getenv(const char* name, char* buffer, size_t* size);
                                                   ^
c_generated_functions.c:1669:29: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
   int x1276 = luv_os_uname(x1275);
                            ^~~~~
./helpers.h:167:24: note: passing argument to parameter 'buffer' here
int luv_os_uname(char *buffer);
                       ^
15 warnings generated.

[benkb]

Do you have Xcode installed?

Yes I have Xcode is installed, but but there is xcodebuild ...

xcodebuild
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance

lovely ....

and there is even a pretty large issue on node-gyp over this: nodejs/node-gyp#569

So with the help of this, dune can build the example:
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

[benkb]

dune exec example/delay.exe
Waiting...           
Hello, world!

[aantron]

Great!