interrupt an IDLE connection manually

[r10s]

interrupting the IDLE command might also be useful when the same IMAP connection is used for doing some jobs (move, markseen, whatever) as well as for IDLE. in this case, when a new job is queued, it is great if the IDLE command is interruptable.

this issue seems to be similar to #121 (cancel idle connection)

just had a look how this is done in libetpan, seems as if they're doing it by writing a single byte to the stream https://github.com/dinhviethoa/libetpan/blob/master/src/data-types/mailstream_cancel.c#L173 (for mac, they use an additional approach, https://github.com/dinhviethoa/libetpan/blob/master/src/data-types/mailstream_cfstream.c#L1183 )

not sure if this is doable in the same way, however.

cc @hpk42

[VP-]

What about making the terminate() function thread-safe and public?

[seijikun]

Wouldn't it be easiest, to make the wait*() methods of imap::extensions::idle::Handle non-consuming?
This would allow something like this:

let mut idle_handle = session.idle().unwrap();
loop {
	idle_handle.wait_timeout(Duration::from_millis(500));
	if should_exit {
		break;
	}
}

[jonhoo]

@seijikun What you're proposing should now be possible with the new wait_while functions introduced by #186. It doesn't quite close out this issue, since there's still the question of being able to cancel at any time (not just at a pre-defined timeout), but that's a much trickier prospect in synchronous code.

[OvermindDL1]

I've been trying to figure out how I can interrupt the IDLE as well, I'm wanting to check if a signal has been set, optimally when it's set via some kind of interrupt, but at least via polling it once every half second or second or so, but it seems the best I can really find with the API that is given (wait_while) is to set a timeout to the 0.5 or 1 second and just tear down and bring up the IDLE wait again, which makes for an incredibly noisy network. I haven't been able to find a way yet to send, say, a timeout to the wait_while via the UnsolicitedResponse to indicate a timeout, or send anything into that callback at all at a repeated time as it seems calling wait_while resends the IDLE command every time it's invoked (also, should wait_inner be calling itself recursively on timeouts endless? That seems stack blowing after a while...). If there were an async version of this then we could select on one or the other but as it is the thread that is handling this seems completely consumed by this non-interruptable wait?

[jonhoo]

Yeah, unfortunately there basically isn't a way to do what you're asking for without async. One thread is blocked on the read from the IDLE connection, and there isn't really a way to interrupt it without introducing a select-like construct, which is basically how you get to async. I do think there may be an argument for just doing an entirely async implementation of imap (there's already async-imap) to support those kinds of use-cases. For the purposes of this crate as it currently stands though, I don't think we'll get to "interrupt arbitrarily".

[vsl-iil]

I do understand that implementing such functionality may be tricky, but I agree that IDLE in its current state isn't that convenient to use. If I want my daemonized program that checks inbox for new mail to shutdown gracefully on SIGTERM, currently I have two options: let it die without sending LOGOUT to the server or don't use IDLE and check the mail manually every minute or less (which is a mess for the network and slower than getting unsolicited messages).
As a solution, maybe we could let the user have a more low-level access and send single IDLE and DONE commands with specialized functions? Something like that:

// separate thread:
while !atomicbool.load(Ordering::Relaxed) {
// sending a single IDLE command
    imap_session.idle().send_once();
    let unread = imap_session.search("UNSEEN");
// setting atomic bool to true and sending anything to recv stops the cycle immediately  
    recv.recv_timeout(Duration::new(29*60, 0));
// sending DONE command
    imap_session.send_done();
}
// saying goodbye
imap_session.logout()?;

It gives a programmer an opportunity to mess up with, for example, properly sending DONE after IDLE, but in the worst case it's not worse than not sending logout, I think.

[jonhoo]

Having some way to a) time out waiting on IDLE (to run other code) and b) send DONE while IDLEing does indeed sound useful! @mordak touched that code most recently, and may be able to give some pointers on how it could be implemented! We'll want to still hold on to the &mut borrow of the Client as long as the IDLE is still going, which I think means we may have to make the IDLE interface such that you call wait on the IDLE handle so that the handle doesn't go away on a time-out. The tricky part is that is that a user could just mem::forget the IDLE handle and then end up with a broken IMAP connection (because DONE wasn't send), but perhaps that's not something we need to worry about.

[mordak]

Sorry for the delay catching up here.

As jonhoo pointed out, the reason that IDLE is uninterruptible is because under the hood it is sitting on a socket read, and they are uninterruptible because the thread is asleep. There is no way around this other than reworking the library to add an async option or similar so the thread can continue while the socket is waiting for data.

You don't want to do this anyway, because you can't do anything else on that socket while the IDLE is running. From the RFC:

The client MUST NOT send a command while the server is waiting for the DONE, since the server will not be able to distinguish a command from a continuation

Even if we want to poll on the socket and do other housekeeping while we're waiting (checking for an exit flag, say), in order to prevent the program from pegging the CPU it will have to sleep at least a little bit, check for the exit flag, poll the socket, and then sleep again, so there is at least some minimal delay between sending the exit command and gracefully shutting down, or receiving a notification from the server and acting on it.

You shouldn't worry about hanging up on the server while it is IDLE. The server has no qualms about hanging up on you:

The server MAY consider a client inactive if it has an IDLE command running, and if such a server has an inactivity timeout it MAY log the client off implicitly at the end of its timeout period.

And your IMAP server is perfectly okay if the client disappears without a graceful LOGOUT while IDLE, it happens all the time. Network changes or interruptions, power outages, program crashes, etc., all mean your IMAP server has to be robust against the client disappearing, and since the IDLE client socket is not writing to the server, there is no worry about inconsistent state or anything. So when your program gets told to exit, just quit and let the socket die.

In my experience, the most efficient way to IDLE is to spin off a worker thread that just remains IDLE all of the time, and every time there is a change sends a message down a channel to a worker thread who actually interacts with the IMAP server and the file system / mail program, etc.. So there are two threads, one sitting on IDLE that is asleep most of the time, and a worker that is sitting on a channel (and is also mostly asleep most of the time) waiting for change notifications from the IDLE thread (or exit messages from the user, etc.) that opens a separate connection to the IMAP server and actually does operations, retrieves messages, etc. This way your program is entirely asleep unless something actually happens, and when something does happen it is instantly noticed and handled because all the threads wake up and do the work as soon as the server sends any message, and then go back to sleep. You can implement a third thread that watches the file system and also sleeps most of the time unless something happens, and have it send messages to the same worker thread. In this scenario the program is asleep almost all of the time and consumes no resources, it is instantly reactive to any changes from the server, and it is almost silent on the network. When you want to exit the program just hangup / kill the IDLE socket - the server will not mind.

So I wouldn't worry about being able to interrupt the IDLE thread, just let it sit there asleep until a notification arrives, and do the real work in another thread that has a separate connection to the server and handles the business of synchronizing the server with the client.