Heartbeat issue

[jibanes]

Since I use centurylink, I often (several times a month) lose my internet connection, because I often work remotely, I wrote a little script that reboots the cable modem when this happens (which runs linux 2.0+busybox) using a little expect script to log in (the cable modem allows login as root without password).

When this happened, 2 days ago, and irclogger was running (for 5 days), the irclogger bot dropped of the irc channels being watched and when the connection was re-established (maybe a few minutes later), the bot never made it back to the channel, and irclogger was still running; on the other end, my irssi session was still running.

My current workaround is to send a sigint signal to irclogger when I run the expect script to reset the cable modem.

Quite possibly, irclogger needs some heartbeat monitor that makes sure that a connection to the irc server is still ongoing, but perhaps also change nick as the irc server might have the other nick in a timeout state (I've noticed that sometimes my nick changes on irssi when this happens). I use irc.freenode.net.

Thank you for looking into it; you might use tcpkill to terminate an ongoing connection to make testing easier; it's a debian/ubuntu package.

[prologic]

So I guess we need some logs of what HAPPENS in this state. Would it be possible for you to run irclogger with the -v option I think? And we'll see an excerpt of the debug logs when this happens :)

[jibanes]

I have a repro, with -v
I'm emailing you a log file. This is what I did:
./irclogger -v -c '#test' -n blahblah irc.freenode.net 6667 2> /tmp/irclogger.log
then I did a few tcpkill -i eth0 port 6667
I did let the first tcpkill last 2 seconds, it terminated irclogger's connection (I did some irc activity on another computer in the same channel)
irclogger reconnected just fine.
Then I did another tcpkill, for longer, like 30 seconds.
the bot never rejoined the channel, and is stuck in a loop.

I'm emailing you the complete log file.

[prologic]

Interesting. It's supposed to:

• reconnect ever 5s on socket error(s)
• reconenct after disconnection

Hmm

[jibanes]

Looks like it happened again a few minutes ago, but this time I didn't lose my network connection, just the irc server bounced; the client didn't come back. But the process was still running.

[prologic]

Yeah okay... I'll do some testing and see what I can work out. It'll be something simple I'm sure :) Meanwhile can you work-around this for now? :)

[jibanes]

did you have a chance to look into this? It's not very easy to make a workaround for this; unless maybe you could exit gracely after a configurable number of retries (-r 10) with (-r 0) for unlimited retries.

[prologic]

Sadly I've only managed to the time to try and test it out once.

I've been so busy of late.

I'll try and have another look tonight :)

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

On Sat, May 9, 2015 at 1:41 AM, jibanes [email protected] wrote:

did you have a chance to look into this?

—
Reply to this email directly or view it on GitHub
#10 (comment).

[jibanes]

No worries, I understand.

[prologic]

Damn I've left this a bit too long :/ Real sorry :(

I'm not sure I've ever seen this problem in the wild myself. The only way I can think of solving this (but adds a lot of complexity to this simple logger) is via a "keep alive" feature such as the one implemented in kdb's keepalive plugin

[prologic]

Also I should point out that irclogger (this project) has been entirely ported to kdb as a plugin where kdb itself is way more feature rich. The only thing is lacks currently is a processor or some kind of UI for the raw logs.