ACP: Unix socket ancillary data v2

[jmillikin]

Proposal

Problem statement

The current unstable support for Unix socket ancillary data (feature unix_socket_ancillary_data) has several known issues and cannot be stabilized in its current form (see comments by @m-ou-se on rust-lang/rust#76915).

@m-ou-se suggested that there needs to be an RFC for Unix socket API ancillary data, which I've started a draft of at rust-lang/rfcs#3430 (rendered).

On Zulip, @pitaj suggested that filing an ACP might also be appropriate.

Motivating examples or use cases

I would like to be able to transfer file descriptors via SCM_RIGHTS ancillary data on platforms that support that functionality (Linux, *BSD, most Unix-ish).

I would also like to be able to obtain platform-specific socket metadata such as Linux's high-resolution packet timestamps.

Solution sketch

The linked RFC 3430 (rendered) contains a draft API for representing ancillary data, including file descriptor ownership and extension points so that third-party libraries can provide platform-specific logic.

I have a local branch that implements that RFC's proposed API. Once the RFC seems to be moving towards stability, I'll add docs + tests to my branch and push it to provide better context as we work out the implementation details.

Alternatives

1. Do nothing, keep the current perma-unstable API until a better solution can be designed.
2. Remove ancillary data support from stdlib and leave it to third-party libraries such as https://github.com/nix-rust/nix.
3. Instead of providing a high-level API, expose more of the underlying BSD sockets API (CMSG_* macros and friends) and let third-party libraries provide wrappers.
4. Expose simple functions for SCM_RIGHTS only, analogous to Python's socket.send_fds() and socket.recv_fds() functions.

None of these are particularly appealing to me, though if anyone does have ideas on an even better API then I'd be happy to see it.

Links and related work

• Tracking Issue for feature(unix_socket_ancillary_data) rust#76915 (tracking issue for unix_socket_ancillary_data)
• unix: Extend UnixStream and UnixDatagram to send and receive file descriptors rust#69864 (original implementation PR for unix_socket_ancillary_data feature)
• Tracking Issue for feature(unix_socket_peek) rust#76923 (a related feature unix_socket_peek that this API would subsume)
• https://blog.cloudflare.com/know-your-scm_rights/ a blog post about SCM_RIGHTS in C, Go, and Rust (via libc).
• https://go.googlesource.com/sys/+/v0.13.0/unix/creds_test.go (tests for ancillary data support in Go stdlib)

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[tmccombs]

Expose simple functions for SCM_RIGHTS only, analogous to Python's socket.send_fds() and socket.recv_fds() functions

Given that this is one of the most common use case for ancillary data, I think it might be worth having an easy to use API for this regardless of whether there is a different more general, lower level API for ancillary data.

[jmillikin]

Given that this is one of the most common use case for ancillary data, I think it might be worth having an easy to use API for this regardless of whether there is a different more general, lower level API for ancillary data.

I think the MessageSender / MessageReceiver helper provides this -- see examples in the RFC for sending/receiving a File.

If you're interested in a one-line solution then the following signatures seem like they'd work:

impl UnixStream {
	pub fn send_fds(&self, bufs: &[IoSlice], fds: &[BorrowedFd]) -> Result<usize>;

	pub fn recv_fds<T>(&self, bufs: &mut [IoSliceMut], fds: &mut T) -> Result<usize>
	where
		T: Extend<OwnedFd>;
}

However, I think such a request would be better as its own ACP. That way this one can remain focused on the handling of ancillary data in general (including OS-control messages). The public APIs are disjoint so there'd be no conflict.

[programmerjake]

	pub fn recv_fds<T>(&self, bufs: &mut [IoSliceMut], fds: &mut T) -> Result<usize>
	where
		T: Extend<OwnedFd>;
}

wouldn't you need to pass in a buffer length for the number of OwnedFd to expect? e.g. some programs only expect 1 FD, but others expect quite a lot, and I don't expect std to be able to pick a sensible default.

[jmillikin]

wouldn't you need to pass in a buffer length for the number of OwnedFd to expect? e.g. some programs only expect 1 FD, but others expect quite a lot, and I don't expect std to be able to pick a sensible default.

The buffer length is provided to AncillaryData::with_capacity() -- as you note it must be large enough to store the received file descriptors, plus any additional expected control messages. See the example in the rendered RFC for an example.

I'm somewhat undecided on how user should be expected to calculate the capacity.

A low-effort but somewhat unergonomic solution is to expose CMSG_SPACE() more-or-less directly, and have the user sum up the space requirements of their expected control messages:

let mut ancillary_capacity = 0;
// ScmRights isn't public right now, but it would need to be to provide just this one helper function.
ancillary_capacity += ScmRights::cmsg_space(10 /* space for 10 FDs */);
ancillary_capacity += os::linux::net::ScmCredentials::cmsg_space();
let mut ancillary = AncillaryData::with_capacity(ancillary_capacity);

Alternatively, it might be possible for std to provide some sort of builder or macro that handles the details, but I haven't been able to find a good API for that. Suggestions welcomed.

[the8472]

but I haven't been able to find a good API for that. Suggestions welcomed.

What issues did you run into?

How about something like

AncillarySize::new()
  .rights(10)
  .credentials()
  .credentials()
  .header_and_typed_payload::<libc::SCM_FOO>()
  .header_and_raw_payload(50)
  .as_usize()

That can combine known and unknown types

[jmillikin]

Sure, that seems like a reasonable approach -- I was nervous about adding yet another public type, but an AncillarySize builder struct would probably work well.

Do any of the Rust libs team have concerns about the overall design of this RFC? If there's no major objections to the new API shape, I'll start polishing up my local prototype branch (add docs, tests, etc) so it can be a PR.

[the8472]

My concerns are pretty much what I've already commented on the rfc

• the start_recvmsg/finish_recvmsg and sendmsg equivalents feel very non-rusty
• ControlMessagesBuf owning an allocation seems quite problematic since network code often wants to work with stack allocations, recycle allocations in weird ways or interpret buffers coming from somewhere else

[joshtriplett]

We discussed this in today's libs-api meeting. We're going to accept this ACP: we do want an API like this. Discussion on the exact details of the API can happen on the RFC.

Thank you!