RedisResilience

This code is derived from Digital Banking github using redistemplate. The focus in this github is on failover and retry when a redis database is not available. This would allow a client to failover from one active/active redis instance to another.

Overview

In this tutorial, a java spring boot application is run through a jar file to support typical API calls to a REDIS banking data layer. A redis docker configuration is included with 2 separate redis containers running at different ports. These are simulating two active/active redis dadtabases. Each database can be easily stopped/started with docker.

Requirements

• Docker installed on your local system, see Docker Installation Instructions.
• Alternatively, can run Redis Enterprise and set the redis host and port in the application.yml file
• When using Docker for Mac or Docker for Windows, the default resources allocated to the linux VM running docker are 2GB RAM and 2 CPU's. Make sure to adjust these resources to meet the resource requirements for the containers you will be running. More information can be found here on adjusting the resources allocated to docker.

Docker for mac Docker for windows

Links that help!

• spring data for redis github
• spring data for redis sample code
• lettuce tips redis spring boot
• spring data Reference in domain
• spring data reference test code
• Resilience4j quickguide
• Resilience4j documentation
• Redis Retry Client
• Workaround for waitDurationInOpenState issue

Technical Overview

This github uses resilience4j circuit breaker and retry mechanisms to make a more resilient client for Spring data redistemplate. A resilience4j circuit breaker is used in a continual loop to do the failover logic while a resilience4j retry is used on client writes to get retry capability. The failover decision is controlled by the circuit breaker callback method.

The spring java code

This is basic spring links

• config-Initial configuration module using autoconfiguration, redistemplate connections, and hard coded resilience4j circuit breaker needed due to a bug. Should be able to define the circuit breaker with application.yml but odd bug with waitDurationInOpenState at 7.1.
• controller-http API call interfaces
• data-code to generate POC type of customer, account, and transaction code
• domain-has each of the java objects with their columns. Enables all the getter/setter methods
• repository-has repository definitions
• service- bankservice doing the interaction with redis

The java code demonstrates common API actions with the data layer in REDIS. The java spring Boot framework minimizes the amount of code to build and maintain this solution. Maven is used to build the java code and the code is deployed to the tomcat server.

Data Structures in use

Getting Started

• Prepare Docker environment-see the Prerequisites section above...
• Pull this github into a directory

git clone https://github.com/jphaugla/redisResilience

• Open terminal and change to the github home where you will see the docker-compose.yml file
  • If docker resources are limited, comment out nodes re2 and re4 in the docker compose file.

docker-compose up -d

NOTE: There is a second docker-compose-consumer.yml file. I was not able to get this to work to run the consumer under the same docker-compose setup. Not sure if it is a resource issue on my mac or a network issue with the second yaml file. For now, the consumer needs to be run outside of docker

• Setup the redis enterprise cluster and create the active/active database
  • if docker resources are limited, comment out the lines for nodes re2 and re4

./setup-2AA.sh
./crdcreate.sh

• bringing up the docker compose will attempt to start the java container called bankapp. However, this will fail because the database was not yet created.
• restart the bankapp and it should work now that cluster is created and database is created

docker-compose start bankapp

To execute the code from command line

(Alternatively, this can be run through intelli4j)

• Compile the code

mvn package

• set the environment variables to match those in docker-compose
• run the jar file.

java -jar target/redis-0.0.1-SNAPSHOT.jar

• Note: parameters for the circuit breaker and retry are in the application.yml. For the circuit breaker, the code reads all the default values such as: resilience4j.circuitbreaker.configs.default.failureRateThreshold. However, the waitDurationInOpenState has an odd error so added waitDurationInOpenStateInt.
• Once the solution is running do a test write

cd scripts
./saveCustomer.sh 
Done%                                                

Test Connection Loop

• Start the connection loop test running with this API call script

./startConnectionLoop.sh

• Now do each of these relatively quickly so, the failover completes before the retry is expired on the write.

docker stop re1
./saveCustomer.sh

NOTE: Because redis1 stopped, the circuit breaker will kick in on the failure of the write to redis in the connection loop. This will call the circuit breaker to go to its callback routine. This callback routine will do a failover once the circuit break opens. At the same time, the client write will retry until successful or maximum retries occur

• re-start redis1

docker start re1

• switch back is a manual process

./switchRedis.sh

Password rotation

• Password rotation tests the ability in an active/active deployment to rotate passwords ensure the same password is used on all instances.
• To ensure all instances have received the password change, a Redis Streams listener is running on each crdb instance.
• When a password change is made, the username, the password, and the timestamp is added to a Redis Stream for the username. Each username will have its own Redis Stream.
• When each CRDB instance listener receives the Redis Streams message, it increments a sorted set score for the paasword member.
• For the getPassword API call, the password score must be the same as the number of crdb instances. So, if there are 5 instances the password score must be 5 to ensure all 5 CRDB instances have the password value
• In case there are multiple passwords with the optimal score, only the most recently updated value is returned using the Redis Stream
• Cleanup API will remove old passwords

Test Password Rotation

• In addition to the normal startup tasks, a consumer needs to be started consuming off the stream in each of the CRDB instances.

export REDIS_URL=redis://localhost:12000
#  user must match the user in password.json in subsequent step
export STREAMS_KEY=STREAM:USER:ralph
cd consumer
mvn package 
./runconsumer.sh

export REDIS_URL=redis://localhost:12001
#  user must match the user in password.json in subsequent step
export STREAMS_KEY=STREAM:USER:ralph
cd consumer
mvn package
./runconsumer.sh

• create a new password

cd ../scripts
./putPassword.sh
./getPassword.sh

• edit the ./scripts/password.json to add a new password
• create the new password and now should get the newerPassword on the get. Check the logs to see what is happening

./putPassword.sh
./getPassword.sh

• now try the step of adding a third password but take down one of the nodes first

docker-compose stop re2
./putPassword.sh
#  make sure this script has targetinstance set to the number of instances
./getPassword.sh

Should return the seconrd and not the third parameter