Classic remoting: half-close on disassociate so a Windows peer doesn't lose our last frames to an RST

[Aaronontheweb]

Closes #8589

Problem

DotNetty 0.7.6 closes a channel with Socket.Shutdown(Both) + Dispose(). If any inbound bytes are still unread at that point, the kernel sends an RST instead of a FIN. A Windows peer that gets an RST throws away everything in its receive buffer that it has not read yet. In #8589 that was sys2's HandOverInProgress, HandOverDone, and ExitingConfirmed. The actor layer wrote every frame into the kernel before the close.

Two paths end in that close: TcpAssociationHandle.Disassociate (per association) and DotNettyTransport.Shutdown (all channels).

Fix

TcpAssociationHandle.Disassociate now closes gracefully:

1. Stop accepting writes. Write returns false once disassociated, because a write after the send side is shut down fails in DoWrite and hard-closes the socket.
2. When the last write has reached the kernel, on the channel's event loop: switch the handler to discard inbound data (no InboundPayload to a stopped listener), turn AutoRead on, and shut down only the send side. The peer reads our frames, then a FIN.
3. Keep reading until the peer's FIN. DotNetty then closes the channel over an empty receive buffer, which sends a FIN, not an RST.
4. A force-close scheduled on the channel's event loop (not the ActorSystem scheduler, which may be shutting down) bounds the wait by akka.remote.flush-wait-on-shutdown (default 2 s).

DotNettyTransport.Shutdown waits up to flush-wait-on-shutdown for the channels that are closing gracefully, then runs its existing force-close loop. A channel registers as draining synchronously, as the first step of Disassociate. Through the protocol transport that happens before the transport shutdown: ActorTransportAdapter.Shutdown stops the protocol manager first, and each ProtocolStateActor disassociates its handle in OnTermination. Channels that are never disassociated (the throttler adapter's, for example) don't make Shutdown wait.

No wire format or public API change.

The half-close uses an alias socket

TcpSocketChannel.ShutdownOutputAsync() doesn't work here. It calls Socket.Shutdown(Send), which clears Socket.Connected, and DotNetty's DoReadBytes treats !Socket.Connected as EOF. The channel then closes on the next read, over whatever is unread, and sends the RST anyway. (Checked: with ShutdownOutputAsync the spec's first test still fails with SO_ERROR = 10058.)

Instead, the send side is shut down through a second Socket built over the same handle (ownsHandle: false, so disposing it doesn't close anything). The original Socket stays Connected, so DotNetty keeps reading until the real FIN.

• The handle comes from one reflection read of DotNetty's protected AbstractSocketChannel.Socket field. A test asserts the field exists, and the transport logs a warning if it's missing.
• The alias sets Blocking = false before Shutdown. Shutdown re-applies the alias's blocking mode to the shared handle, and on Windows a Socket built from a handle assumes blocking.
• Any failure falls back to today's CloseAsync() and logs a warning.

Where this differs from the plan in #8589

• Every Disassociate takes the graceful path, not just Shutdown-reason ones. The handle does not know the reason. The cost for quarantine and failure paths is at most flush-wait-on-shutdown before the socket closes.
• ChannelInactive and ExceptionCaught still notify the listener while draining. Only ChannelRead discards. Disassociated is already IDeadLetterSuppression, and DotNettyTransportShutdownSpec relies on the local listener getting Disassociated after a local disassociate.

Tests

New DotNettyGracefulCloseSpec (Akka.Remote.Tests/Transport) drives TcpTransport against a raw Socket peer. Unless a test says otherwise, no ReadHandlerSource is set, so the peer's bytes sit unread on our side:

1. The peer sends 1 KB. We write a frame and disassociate. The peer reads the frame, then EOF, and its SO_ERROR stays 0. After the peer closes, our channel closes within 1 s. On dev this fails with SO_ERROR = 10058 (the reset that loses frames on Windows; Linux still delivers the bytes, so the error code is the signal).
2. TLS on (reads enabled so the handshake can finish; the peer's traffic after our disassociate has to drain through TlsHandler): the peer reads our frame, then EOF, with no reset.
3. A peer that never closes: with flush-wait at 300 ms, our channel force-closes within 1.5 s.
4. Shutdown() returns within 1.5 s (flush-wait 300 ms) while a drain is stuck.
5. Both sides disassociate at once, with frames in flight: both channels close within 1 s, and nobody logs a connection reset.
6. Two ActorSystems with an open association and flush-wait 5 s: Terminate() finishes in under 3 s (about 130 ms locally), so a graceful close that stalls on flush-wait shows up.
7. Write returns false after Disassociate (fails on dev).
8. DotNetty's AbstractSocketChannel.Socket field exists, so a DotNetty upgrade can't turn the fix off unnoticed.

Windows CI is the real check

This was developed and tested on Linux. The loss in #8589 happens on Windows, and two parts of this change are Windows-specific in ways I can't check locally:

• The alias socket. On Windows, a Socket built from a handle assumes blocking mode, which Blocking = false works around. Disposing a non-owning alias doesn't close the handle. I read both behaviors from the .NET source; neither has run on Windows yet.
• Shutdown(Both) after the half-close. DotNetty's final close still calls Socket.Shutdown(Both) on the original socket. On Linux, .NET ignores ENOTCONN there, with a comment that this "matches Winsock behavior", so I expect Winsock to return success too. If it did throw, DotNetty's DoClose0 still completes the channel's close future, and only the handle's Dispose would move to the finalizer.

The Windows unit-test lane on this PR is the real check.

[Aaronontheweb]

LGTM