Review and Document Findings for Redis cluster not starting gracefully when there is disruption in openshift

[guru-aot]

Redis cluster not starting gracefully when there is disruption in openshift, the main reason which we found was, the restart make the slave node to be in intermediate position as they are not finding the master node.

https://github.com/bcgov/SIMS/wiki/DevOps-and-Running-the-Application#redis-cluster-failing-to-recover

Additional Context

• The only option we are following right now is by deleting the existing redis cluster using the make delete-redis command and re initializing and starting a new one.
• The problem on the above approach is, its deletes the PVC of the redis and the queues that are waiting will all be lost.

Acceptance Criteria

• Find a solution either by starting the redis in standalone or any configuration to restart redis geacefully during any disruption in the environment. (review and document)
• Investigate in the community as this is a common problem (rocket chat)
• Also check why there is a restart of workers and API needed while this restart of redis happens, find if we can create a depends on in redis restart to automatically restart API and workers.
• Investigate the automatic recovery of redis cluster if the cluster is setup with only 3 masters (3pods) unlike the current setup with 3 masters and 3 slaves(6pods)

[cditcher]

This appears to be a common problem. For reference: https://chat.developer.gov.bc.ca/channel/redis

[dheepak-aot]

Added an AC as per discussion in dev call

[guru-aot]

Restart of api and workers pods are not necessary, but restarting the queue consumers is required.

Also wiki has been appropriately updated
https://github.com/bcgov/SIMS/wiki/DevOps-and-Running-the-Application#redis-cluster-failing-to-recover

[andrewsignori-aot]

• Also check why there is a restart of workers and API needed while this restart of redis happens, find if we can create a depends on in redis restart to automatically restart API and workers.

What is the technical reason to restart queue-consumers? Is there something that can be changed to avoid manual intervention?

[guru-aot]

• starting

I am not sure, the queue consumers are not able to connect with the Redis and the dashboard was not loading. Once we restart we were able to recover the dashboard.