'testRaceNewRequestsVsShutdown' fails frequently on macOS

[glbrntt]

More specifically the opened connections count invariant assertion fails here. We expect to have no open connection at this point, however, I saw a negative connection count each time it failed.

I noticed that the connections were often released multiple times when in the .closed state here.

I hit this against c9a9bf061d713c91ee9974fa8a6afe413acfd0e9 on macOS (Xcode 12 beta) and was unable to reproduce it against the swift:5.2 Docker image.

[Lukasa]

Also got a failure in CI in #335:

Test Case 'HTTPClientTests.testRaceNewRequestsVsShutdown' started at 2021-01-25 09:31:21.753
Assertion failed: file /code/Sources/AsyncHTTPClient/ConnectionsState.swift, line 82
Current stack trace:
0    libswiftCore.so                    0x00007f18a8765db0 swift_reportError + 50
1    libswiftCore.so                    0x00007f18a87d7f60 _swift_stdlib_reportFatalErrorInFile + 115
2    libswiftCore.so                    0x00007f18a84ecdfe <unavailable> + 1383934
3    libswiftCore.so                    0x00007f18a84eca07 <unavailable> + 1382919
4    libswiftCore.so                    0x00007f18a84ed032 <unavailable> + 1384498
5    libswiftCore.so                    0x00007f18a84eb2d0 _assertionFailure(_:_:file:line:flags:) + 520
6    async-http-clientPackageTests.xctest 0x000055ba19b1c5a4 <unavailable> + 628132
7    async-http-clientPackageTests.xctest 0x000055ba19b102e6 <unavailable> + 578278
8    async-http-clientPackageTests.xctest 0x000055ba19b10449 <unavailable> + 578633
9    libswiftCore.so                    0x00007f18a8767d1b <unavailable> + 3984667
10   async-http-clientPackageTests.xctest 0x000055ba19f8e362 <unavailable> + 5288802
11   libswiftCore.so                    0x00007f18a8767d1b <unavailable> + 3984667
12   async-http-clientPackageTests.xctest 0x000055ba19f8de62 <unavailable> + 5287522
13   libswiftCore.so                    0x00007f18a8767d1b <unavailable> + 3984667
14   async-http-clientPackageTests.xctest 0x000055ba1a011935 <unavailable> + 5826869
15   libswiftCore.so                    0x00007f18a8767d1b <unavailable> + 3984667
16   libswiftCore.so                    0x00007f18a875f210 swift_arrayDestroy + 59
17   libswiftCore.so                    0x00007f18a84cf010 _ArrayBufferProtocol.replaceSubrange<A>(_:with:elementsOf:) + 2350
18   libswiftCore.so                    0x00007f18a85048e0 ContiguousArray.replaceSubrange<A>(_:with:) + 213
19   async-http-clientPackageTests.xctest 0x000055ba1a00b855 <unavailable> + 5802069
20   async-http-clientPackageTests.xctest 0x000055ba19f7b3a2 <unavailable> + 5211042
21   async-http-clientPackageTests.xctest 0x000055ba19f7bb9f <unavailable> + 5213087
22   async-http-clientPackageTests.xctest 0x000055ba19f82ab3 <unavailable> + 5241523
23   async-http-clientPackageTests.xctest 0x000055ba19f7c1af <unavailable> + 5214639
24   async-http-clientPackageTests.xctest 0x000055ba1a0667d1 <unavailable> + 6174673
25   async-http-clientPackageTests.xctest 0x000055ba1a06a0f9 <unavailable> + 6189305
26   async-http-clientPackageTests.xctest 0x000055ba1a06a2a9 <unavailable> + 6189737
27   libpthread.so.0                    0x00007f18a81836db <unavailable> + 30427
28   libc.so.6                          0x00007f18a62bf6e0 clone + 63
Exited with signal code 4

[Lukasa]

And then it hanged in the next run on 5.3.

[Lukasa]

The call stack above points at HTTP1ConnectionProvider.ConnectionsState.assertInvariants, line 82:

    func assertInvariants() {
        assert(self.waiters.isEmpty)
        assert(self.availableConnections.isEmpty)
        assert(self.leasedConnections.isEmpty)
        assert(self.openedConnectionsCount == 0)
--->    assert(self.pending == 0)
    }

Of course this isn't all that helpful because we need to know how the heck we got here. The above call stack isn't much use to us to diagnose that. Thankfully the only call to assertInvariants is in the HTTPConnectionProvider.deinit.

[weissi]

CC @artemredkin

[artemredkin]

I'll look into this issue

[weissi]

CC @artemredkin / @fabianfett

[fabianfett]

Implementation replaced in #427. 100 consecutive runs on macOS didn't show this issue again.