box: finish client fibers on shutdown

[nshy]

In the process of graceful shutdown it is convenient to first finish all client (non system) fibers. Otherwise we should be ready for any subsystem to handle request from client fiber during or after subsystem shutdown. This would make code more complex.

We first cancel client fibers and then wait for their finishing. The fiber may not respond to cancel and hang which cause shutdown hang but this is the approach we choose for iproto shutdown already.

Note that as a result of this approach application will panic if it is shutdown during execution of initialization script (in particular if this script is doing box.cfg).

There are changes in application/test to adopt to client fibers shutdown:

• make code cancellable (only to pass existing tests, we did not investigate all the possible places that should be made such).

• make console stop sending echo to client before client fibers shutdown. Otherwise as console server fiber is client one we will send message that fiber is cancelled on shutdown which breaks a lot of existing tests. This approach is on par with iproto shutdown.

• some tests (7743, replication-luatest/shutdown, replication/anon, replication/force_recovery etc etc) test shutdown during execution of init script. Now panic is expected so change them accordingly.

• some tests (8530, 5430, errinj_vylog) use injection that block client fiber finishing. In that tests we don't need graceful shutdown so let's just kill tarantool instead.

• we change test in vinyl/errinj for gh-3225. We don't really need to check when vinyl reader is blocked as it executes small tasks (we assume reading syscall will not hang). Also change test for vinyl dump shutdown by slowing dump down instead of blocking it entirely. This is required to finish in time client fibers in the test.

• other similar changes

Also we can drop code from replication shutdown which is required to handle client requests during/after shutdown.

Part of ##8423

[coveralls]

coverage: 86.927% (+0.003%) from 86.924%
when pulling ef70297 on nshy:client-fibers-shutdown
into 3fcd83e
on tarantool:master.

[nshy]

I also added this hunk to fix memleak with injections.

--- a/src/box/memtx_engine.cc
+++ b/src/box/memtx_engine.cc
@@ -1028,7 +1028,7 @@ checkpoint_f(va_list ap)
                return -1;
        }

-       struct mh_i32_t *temp_space_ids = mh_i32_new();
+       struct mh_i32_t *temp_space_ids;

        say_info("saving snapshot `%s'", snap->filename);
        ERROR_INJECT_WHILE(ERRINJ_SNAP_WRITE_DELAY, {
@@ -1040,6 +1040,7 @@ checkpoint_f(va_list ap)
        });
        ERROR_INJECT(ERRINJ_SNAP_SKIP_ALL_ROWS, goto done);
        struct space_read_view *space_rv;
+       temp_space_ids = mh_i32_new();
        read_view_foreach_space(space_rv, &ckpt->rv) {
                FiberGCChecker gc_check;
                bool skip = false;