Support waiting on POLLPRI events on Linux

[ColonelThirtyTwo]

Is your feature request related to a problem? Please describe.
/proc/self/mounts raises a priority event when there is a change in mount points (something is mounted or unmounted). There are probably other similar system-related files that do this as well. There appears to be no way of waiting for these in Tokio.

Describe the solution you'd like
Add a Priority variant to tokio::io::Interest and implement support for it in tokio::io::unix::AsyncFd.

Describe alternatives you've considered
Polling for changes manually. Writing my own select/poll loop in a blocking thread.

Additional context
None

[Noah-Kennedy]

I think it may be worthwhile for us to think about support for other event types in epoll and kqueue in general actually.

[Noah-Kennedy]

@ColonelThirtyTwo A workaround for this is that an epoll fd is itself pollable, so you can create an epoll for watching event types we do not support yet, and put it in an AsyncFd to get notified when it has events ready to be read.

[ColonelThirtyTwo]

Apparently mio does listen for priority events if you use the readable interest, though you'll get a single read ready event at the beginning of watching.

[folkertdev]

as a datapoint, this is what we do in ntpd-rs:

We have an existing udp socket. Based on that socket we create an additional "exceptional condition" socket that becomes readable when there is something in the error queue

/// Turn a C failure (-1 is returned) into a rust Result
pub(crate) fn cerr(t: libc::c_int) -> std::io::Result<libc::c_int> {
    match t {
        -1 => Err(std::io::Error::last_os_error()),
        _ => Ok(t),
    }
}

pub(crate) fn exceptional_condition_fd(
    socket_of_interest: &std::net::UdpSocket,
) -> std::io::Result<AsyncFd<RawFd>> {
    let fd = cerr(unsafe { libc::epoll_create1(0) })?;

    let mut event = libc::epoll_event {
        events: libc::EPOLLPRI as u32,
        u64: 0u64,
    };

    cerr(unsafe {
        libc::epoll_ctl(
            fd,
            libc::EPOLL_CTL_ADD,
            socket_of_interest.as_raw_fd(),
            &mut event,
        )
    })?;

    AsyncFd::new(fd)
}

Then to consume, we can await the exceptional condition fd to become readable:

async fn fetch_send_timestamp(&self, expected_counter: u32) -> io::Result<NtpTimestamp> {
    trace!("waiting for timestamp socket to become readable to fetch a send timestamp");
    loop {
        // Send timestamps are sent to the udp socket's error queue. Sadly, tokio does not
        // currently support awaiting whether there is something in the error queue
        // see https://github.com/tokio-rs/tokio/issues/4885.
        //
        // Therefore, we manually configure an extra file descriptor to listen for POLLPRI on
        // the main udp socket. This `exceptional_condition` file descriptor becomes readable
        // when there is something in the error queue.
        let mut guard = self.exceptional_condition.readable().await?;
        match guard.try_io(|_| fetch_send_timestamp_help(self.io.get_ref(), expected_counter)) {
            Ok(Ok(Some(send_timestamp))) => {
                return Ok(send_timestamp);
            }
            Ok(Ok(None)) => {
                continue;
            }
            Ok(Err(e)) => {
                warn!(error = debug(&e), "Error fetching timestamp");
                return Err(e);
            }
            Err(_would_block) => {
                trace!("timestamp blocked after becoming readable, retrying");
                continue;
            }
        }
    }
}

It would be really nice if we could off-load this complexity onto tokio. Until then, maybe this code is helpful to someone.

[folkertdev]

It looks like mio recently added the PRIORITY interest https://docs.rs/mio/0.8.6/src/mio/interest.rs.html#49. When tokio updates its version of mio, tokio could expose this interest (for linux and android, anyway).

To me, that sounds like a really nice solution for this issue. Is there anything against exposing Interest::PRIORITY?