Detach execution context scheduler from running thread during blocking syscall

[ysbaddaden]

Some syscalls can block the current thread in certain circumstances, for example:

• open(2) when opening a FIFO, pipe or character
  device until another end is connected (from another thread or process);
• getaddrinfo(3) until a DNS response (or error, or timeout) is received.

This patch introduces a mechanism to declare the scheduler as "doing a syscall" which the monitor thread (SYSMON) can detect on its next iteration and will try to move the scheduler to another thread, so that only the fiber doing the syscall will be blocked, and the other fibers can be resumed.

Usually, the syscall should terminate before the monitor thread notices (for example opening a regular file), so the impact on performance is an atomic STORE + atomic CAS per syscall. At worst, a thread will be blocked for 10ms (SYSMON frequency). For example the updated opening FIFO file spec takes ~11ms to complete.

It works for the MT execution contexts and the ST context. It doesn't invalidate the ST guarantee that fibers in the context will never run in parallel: the blocked fiber is blocked on a syscall and will be re-enqueued immediately after the syscall has completed; also the syscalls don't invoke callbacks that would execute crystal code, so AFAICT fibers still won't run in parallel (please correct me if I'm wrong).

NOTES

The isolated context expects to block, so the #syscall(&) method is a no-op there.

There are probably other blocking syscalls that we might want to consider. For example, reading from STDIN on Windows could be greatly simplified.

Another example is flock that is currently retried every 100ms when it doesn't block the current thread. We might want to be able to actively detach a scheduler when calling #syscall(&), so we could try once (non-blocking) then on failure detach the scheduler and try again (blocking) without waiting for SYSMON to notice.

EXAMPLE

The following example blocks the current thread, yet the spawned fiber keeps ticking every second. Remove the Fiber.syscall wrapper, and the fiber won't even start!

# bin/crystal foo.cr -Dpreview_mt -Dexecution_context

spawn do
  loop do
    sleep 1.second
    puts "tick"
  end
end

Fiber.syscall do
  Thread.sleep(5.seconds)
end

FOLLOW UP

We plan to use this in the future to rework and simplify use cases in the stdlib. For example:

• Polling event loops could support blocking file descriptors, so we could stop setting O_NONBLOCK on standard descriptors (shared), including pipes to spawned processes (Crystal's duplicated stdios are broken #16353).

• Same on Windows where console streams don't support OVERLAPPED (Console streams are blocking on Windows #14576, Emulate non-blocking STDIN console on Windows #14947).

---

THREAD POOL

This PoC also introduces a pool of threads. It changes the behavior of threads: we don't start a thread to run a specific scheduler run loop, but each thread now has its own inner loop that basically switches to a scheduler loop then switches back to its inner loop to sleep.

The benefit of the global thread loop is that threads are kept around instead of being created and thrown away. If you regularly spawn an isolated fiber, it will likely keep reusing the same thread(s). Threads still eventually shutdown after some inactive time (configurable) except for the main thread (we need to keep the main fiber alive).

A potential evolution will park MT threads into the thread pool, instead of keeping them tied to the MT context, so they can be reused by any context that needs parallelism, or to boot a new isolated fiber or ST context.

Extracted to #15885.

POTENTIAL ISSUES

I got one segfault in a gc call nested a libxml2 callback in one early run of the std specs (with -Dpreview_mt -Dexecution_context) but I couldn't reproduce it after fixing different issues in the PR.

Maybe it was a fluke (because of the bugs), or maybe it was just a regular MT issue with libxml2, or maybe sysmon moved the scheduler from the main thread to another thread then resumed a fiber doing something in libxml2, and the global thread local state couldn't be found?

This is the already known MT issue we have with libxml2. What's new is that the segfault might start happening in a ST environment 😢

☝️ expected to have been fixed by #15899 (and #15906).

[ysbaddaden]

Rebased from master to bring #15885 along with a few fixes. Ready for review.

[ysbaddaden]

I finally got a trace in gdb:

#4  0x00005a072d66a408 in scheduler (self=0x7b863afc6c00) at /home/julien/work/crystal-lang/crystal/src/crystal/system/thread.cr:82
#5  0x00005a072d67730e in current () at /home/julien/work/crystal-lang/crystal/src/fiber/execution_context/scheduler.cr:6
#6  0x00005a072d706e19 in enqueue (self=0x7b866f75dea0, fiber=0x7b863afc4600)
    at /home/julien/work/crystal-lang/crystal/src/fiber/execution_context/parallel.cr:250
#7  0x00005a072d6ea7b5 in open (self=0x7b866f77ce70, path=0x7b863b00ac80, flags=577, permissions=420,
    blocking=<error reading variable: Attempt to dereference a generic pointer.>)
    at /home/julien/work/crystal-lang/crystal/src/fiber/execution_context/scheduler.cr:106
#8  0x00005a072d6e2c0a in open (filename=0x7b863b00ac80, mode=0x5a072d2b6090, perm=64262,
    blocking=<error reading variable: Attempt to dereference a generic pointer.>)
    at /home/julien/work/crystal-lang/crystal/src/crystal/system/unix/file.cr:9
#9  0x00005a072d6d5996 in new_internal (filename=0x7b86327ffba0, mode=0x5a072d2b6090, perm=420,
    encoding=<error reading variable: Attempt to dereference a generic pointer.>,
    invalid=<error reading variable: Attempt to dereference a generic pointer.>, blocking=...)
    at /home/julien/work/crystal-lang/crystal/src/file.cr:176
#10 0x00005a072d6dc6e1 in write (filename=0x7b863b00ac80, content=..., perm=420, encoding=758155224, invalid=10063,
    mode=0x5a072d2b6090, blocking=989900027) at /home/julien/work/crystal-lang/crystal/src/file.cr:536
#11 0x00005a072d6dc5ff in write (filename=0x7b863b00ac80, content=...) at /home/julien/work/crystal-lang/crystal/src/file.cr:613
#12 0x00005a072ec92a3e in update_bitcode_cache (self=0x7b863af9bd90)
    at /home/julien/work/crystal-lang/crystal/src/compiler/crystal/compiler.cr:1039
#13 0x00005a072ec92694 in compile (self=0x7b863af9bd90, isolate_context=true)
    at /home/julien/work/crystal-lang/crystal/src/compiler/crystal/compiler.cr:997

The monitor thread detached the scheduler from the thread (it noticed it was doing a syscall) and the thread should now be a bare thread. We call Fiber::ExecutionContext::Parallel#enqueue that tries to do a local enqueue which tries to call Fiber::ExecutionContext::Scheduler.current that raises (it's nil).

The thing is, the current thread is now a bare thread, so the condition Fiber::ExecutionContext.current? == self should be nil == self and thus false. The debugger tells me that both are NULL pointers (aka nil).

Theory A: I don't use atomic ops to set Thread#execution_context or Thread#scheduler so maybe open finishes without yielding the CPU time, and maybe it keeps a ref to Thread#execution_context.

Theory B: the thread is being detached by the monitor at the same time the not-yet-bare thread tries to enqueue the fiber... and that's super dangerous: we must ALWAYS do a safe enqueue to the global queue 💣 💥 🤦

[ysbaddaden]

Theory B appears to have been correct. CI doesn't reproduce anymore.