make resource failure when cc parallel is enabled

[NobodyXu]

Since cc >=1.0.80, we set the jobserver inherited to O_NONBLOCK, make probably isn't designed for this, which is why it failed.

I will open a new ticket in cc and fix it.

Originally posted by @NobodyXu in rust-lang/cargo#13476 (comment)

[NobodyXu]

To fix this error, we have to clear O_NONBLOCK before spawning, which is ok and easy to do, given that the parallel version of cc::Build::compile_objects is single-thread only.

We can set the O_NONBLOCK only when acquiring the token.

The error in the linked issue happens, because cc set O_NONBLOCK on the pipe inherited, and then the build.rs spawns a make process that inherits the pipe with O_NONBLOCK set, when the pipe is exhausted, instead of blocking the make, it returns EAGAIN which make is not designed to handle.

It's a bit more expensive given that 2 more syscalls is issued but should be ok.

Ideally I want to find a way to avoid setting O_NONBLOCK at all, but I can't find a race-free way to do it.

Using select to get ready or not or using ioctl to get number of bytes available in pipe is not atomic.

[NobodyXu]

I just realized that setting and clearing O_NONBLOCK is racy because build.rs could have multiple threads and spawning new processes in parallel to cc compilation.

I've discovered pthread_atfork, however it doesn't work for vfork or raw syscall.

So I'm crossing my fingers that newer version of make fixes this.

Otherwise, I could submit a patch to jobserver-rs to always use the new named pipe, which does not trigger this bug, while passing the fds for backwards compatibility.

E.g. MAKEFLAGS='--jobserver-auth=fifo:/path/to/fifo --jobserver-fds=3,5'

[NobodyXu]

Closing this in favor of jobserver issue

[NobodyXu]

Reopen this since I think it will be faster to apply the quick fix to cc, updating jobserver to use fifo will take quite some time.

Even when jobserver is updated to fifo, it's still possible for make to fail if the jobserver is created by an older version of make that uses annymous pipe instead of named fifo.

[NobodyXu]

It's unfortunate that we don't have anyway to clone underlying file descriptor so that the new fd has an independent state from the original to let us set O_NONBLOCK without affecting the original fd.

[Amanieu]

At least on Linux you can open /dev/fd/$FD to open a separate FD pointing to the same underlying file.

[NobodyXu]

@Amanieu If you are ok with it, I can add this as workaround on Linux and then fallback to setting O_NONBLOCK only in Build::compile_objects on other unix OSes.

[the8472]

Note that since gnu make 4.3 they did switch to nonblocking pipes. Though due to an optimization in the linux kernel blocking pipe reads should actually perform better.
jobserver-rs accepts both approaches.

Edit: Pipes, not sockets

[NobodyXu]

Note that since gnu make 4.3 they did switch to nonblocking sockets.

Thanks, but according to @nadenf make 4.4.1 still doesn't work, which is strange

Makes me suspect if I got this wrong.

[NobodyXu]

I verified in cargo-binstall that updates to make 4.4.1 does fix this error.