Recurring reminder silently stops working at some point

[piotr-wegert]

Summary

Recurring reminder silently stops working at some point

Steps to reproduce

Set up a recurring reminder on a public channel e.g. /remind ~test_channel "daily" every weekdays at 9:00 am

Expected behavior

Reminder works indefinitely.

Observed behavior (that appears unintentional)

Reminder works fine for some time. At some point (hard to determine exactly when) it silently stops working. There is no error in the UI and no trace in the logs.

Server configuration

Operating system: CentOS Linux release 7.6.1810 (Core), db: postgres

Mattermost version: 5.12.0

Remind version: 0.4.3

Updated from an older Remind bot version or fresh install:
Updated from an earlier version

Additional context

So far haven't seen anything suspicious in the logs. At this point I'm also unable to determine when the plugin stops working. I'm running some tests so might have additional details later.

[scottleedavis]

Are you able to verify if v0.4.4 has the behavior as well?

[piotr-wegert]

We did try 0.4.4 but got the following error "This plugin failed to start. Check your system logs for errors." and reverted back to 0.4.3.

[scottleedavis]

Ah that's right. 0.4.4 needs 5.14. Any fix I do will also need 5.14 or later

[piotr-wegert]

Thanks Scott! We're planning to bump MM to the latest version this week.

[piotr-wegert]

So our reminder stopped working again. It was set up on October 24th and didn't fire up today morning. I can't find anything meaningful in the logs.

Could you please advise?

[piotr-wegert]

Stopped working again. Unfortunately I can't see any pattern.

[hullin]

Here, all reminders regularly stop working after every update of Mattermost. They are still listed but don't fire any longer.

Using mattermost-plugin-remind - 0.4.4 under docker-compose version of Mattermost 5.19.0 under Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

[stevekerrison]

Here, all reminders regularly stop working after every update of Mattermost. They are still listed but don't fire any longer.

Using mattermost-plugin-remind - 0.4.4 under docker-compose version of Mattermost 5.19.0 under Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

Same here.

[scottleedavis]

Hi All,

Unfortunately, I am completely booked with my current position and do not have time to debug and fix this.

Would Mattermost be open to supporting this plugin @levb ? I would be happy to transfer ownership.

[levb]

@scottleedavis @jfrerich Let's do this. Jason can help with the intake process.

[JohannesK142]

Hi,
our company is using Mattermost Enterprise Edition E10 running on Debian stretch / mysql.

I can confirm the bug as well for Mattermost 5.20.1 and Remindbot 0.4.4.

A fix would be appreciated.

[anarute]

Hello,

We are also facing the same issue. We couldn't determine yet exactly when it occurs, it doesn't follow a pattern, nothing on the logs.
We are using MM 5.18.0 and Remindbot 0.4.4

[scottleedavis]

@hanzei managing a global team is hard work! I don't have time for this fix... do you perhaps? :)

[hanzei]

@scottleedavis I feel you 😉 I'm also low on time at the moment. I will take a look once things are slowing down a bit.

[anarute]

@scottleedavis do you have any clue of where to look to fix this? I can maybe try to help, but I have no idea where to start

[scottleedavis]

@anarute I am not clear where this issue is rooted from.

I had a guess that it was timer inaccuracy in the scheduler, which wouldn't trigger the event... however a timer inaccuracy doesn't seem highly plausible (to me) as variations in local time on server doesn't seem likely in the scale of seconds. (though... who am I to know this in all env's?)
https://github.com/scottleedavis/mattermost-plugin-remind/blob/master/server/scheduler.go#L314

I also thought perhaps somehow the reminder wasn't stored after a 'reschedule' happened in a defer.
https://github.com/scottleedavis/mattermost-plugin-remind/blob/master/server/reminder.go#L260

It could be also, something in coding error on scheduling recurring reminders.
https://github.com/scottleedavis/mattermost-plugin-remind/blob/master/server/scheduler.go#L11

@lieut-data Had ideas of completely redoing the scheduler, and could perhaps solve this root cause issue with refactoring instead of monkey patching the current approach.

👇
👉 #143. 👈
☝️

[scottleedavis]

https://www.linkedin.com/posts/skawtus_remind-plugin-activity-6662399044964401152-aCZw

[JohannesK142]

On my machine I observed that it was low on RAM and another process was killed by the kernel scheduler... Maybe the remind bot also needs some protection against that one of its processes is externally killed...

[scottleedavis]

@levb @lieut-data ?

[hanzei]

The cluster package allows the scheduling of jobs in a concurrency save way. Using that package could greatly help with reducing bugs related to the reminder scheduler.

Integration this package would be a good isolated test, maybe even as a Help Wanted.

[scottleedavis]

The cluster package allows the scheduling of jobs in a concurrency save way. Using that package could greatly help with reducing bugs related to the reminder scheduler. Integration this package would be a good isolated test, maybe even as a Help Wanted.

Open bug bounty. I'll send a $10 amazon gift card for an approved PR.

@hanzei would you help review the PR too?

[scottleedavis]

@levb @lieut-data are also invited to review a PR.

[hanzei]

Sure, I'm happy to review the PR.

[scottleedavis]

I will gift a $50 amazon gift card to the first person who completes this PR.
And some Mattermost stickers.

[Jacq]

I did not found a similar issue for my scenario, but may be is related to this.
In my case the remind feature stops working each time I upgrade mattermost to a new version.
Then I have to setup a new reminder and it works again for the new ones but not for the older ones which are never reminded.
My reminders are reminder at a specific hour each working day for a given channel.
Cheers,
Jacq

[alexander-potemkin]

Not working reliable for me as well. Would love to have this functionality in core of mattermost, like it's core of slack

[Philmod]

I'm guessing the reminders don't survive a Mattermost restart? We deployed it on Kubernetes, so the pod(s) are recycled from time to time.

Is there any way the plugin could store these reminders outside of the process?

[DummyThatMatters]

Same here. Can you guys tell if this happens on plugin restart? Or maybe server restart?

[iamkhalidbashir]

need fix for this