Error while shrinking a cluster

[arielzach]

Hello,

I have a cluster in Elasticache and encountered this error while scaling it down: "redis client is closing or unable to connect redis"

This is the way I am connecting:

client, _ := rueidis.NewClient(rueidis.ClientOption{
		InitAddress: []string{"clustercfg.xxxxx.yyyyy.use1.cache.amazonaws.com:6379"},
		TLSConfig: &tls.Config{
			MinVersion: tls.VersionTLS12,
		},
		MaxFlushDelay: 0,
		Username:      "user",
		Password:      "pass",
		DisableCache:     true,
		ConnWriteTimeout: 2 * time.Second,
		DisableRetry:     false,
	})

client.Do(ctx, client.B().Setex().Key("key").Seconds(120).Value("value").Build()).Error()

I am also seeing these errors when adding a new node, after a while (long), the client stops experiencing errors, the weird thing is that not all requests end in error, but many of them do.

Do I need any additional configuration? Am I doing something wrong? Thank you.

[rueian]

Hi @arielzach,

One thing to mention is that you should not ignore the error returned from rueidis.NewClient. Besides that, your configuration looks good to me.

rueidis uses CLUSTER SLOTS to consult key slot ownerships. From my personal experience, Elasticache really takes a long time, maybe longer than 5 minutes, to return an updated to-date CLUSTER SLOTS response.

I am afraid what you experienced was an expected behavior: Elasticache respond with staled CLUSTER SLOTS responses for a long time and led rueidis to connect to an old and closed redis node.

[arielzach]

Thank you for the response. We have conducted tests with go-redis, and this issue does not occur (it is transparent). Is it possible to handle it in the same way? We do not want to use go-redis due to a bug that generates multiple new connections, which affects the cluster. Rueidis has worked very well in the tests we have conducted. For example, Lettuce (Java) allows filtering of nodes that are not responding correctly. Is it possible to add this functionality or deal with this problem in a better way?

The problem also arose when I enlarged the cluster, do you know what could be the reason?

[rueian]

Hi @arielzach,

I have just also conducted a small test with ElastiCache with the below configuration:

and with the following program:

package main

import (
	"context"
	"log"
	"strconv"
	"time"

	"github.com/rueian/rueidis"
)

func main() {
	c, err := rueidis.NewClient(rueidis.ClientOption{
		InitAddress: []string{"test1.qybrkz.clustercfg.apne1.cache.amazonaws.com:6379"},
	})
	if err != nil {
		panic(err)
	}
	defer c.Close()

	do := func(i int) error {
		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
		defer cancel()
		return c.Do(ctx, c.B().Set().Key(strconv.Itoa(i)).Value(strconv.Itoa(i)).Build()).Error()
	}

	for i := 0; true; i++ {
		if err := do(i); err != nil {
			log.Printf("SET %d FAIL %v\n", i, err)
			continue
		}
		if i%1000 == 0 {
			log.Printf("SET %d SUCC\n", i)
		}
	}
}

During my test on the Add shards and Remove shards operations,

The above program didn't log out any error.

So I think I may need more details for further investigation. Would you mind providing instructions and code to reproduce your experiments? I am also considering releasing a debug build that will log out internal actions and errors of a cluster client if you are willing to provide those logs.

[rueian]

Hi @arielzach,

I just published the v1.0.0 release (also changed the module name under redis) and it will delay connection cleanup a little bit, instead of immediately, after finding those IP addresses are missing from CLUSTER SLOTS responses. I think it may reduce the number of client is closing or unable to connect redis you saw. Would you like to try it?

[arielzach]

Hello, this is the code I used for testing:

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"math/rand"
	"strings"
	"time"

	"github.com/rueian/rueidis"
)

func main() {
	ctx := context.Background()
	client := getRueidisClient()

	go control(ctx, client)
	for i := 1; i <= 18; i++ {
		go set(ctx, client)
	}

	time.Sleep(6 * time.Hour)
}

func getRueidisClient() rueidis.Client {
	client, _ := rueidis.NewClient(rueidis.ClientOption{
		InitAddress: []string{"clustercfg.test1.pxoqnz.use1.cache.amazonaws.com:6379"},
		TLSConfig: &tls.Config{
			MinVersion: tls.VersionTLS12,
		},
		MaxFlushDelay:    0 * time.Microsecond,
		Username:         "user",
		Password:         "pass",
		DisableCache:     true,
		ConnWriteTimeout: 2 * time.Second,
		DisableRetry:     false,
	})

	return client
}

func set(ctx context.Context, client rueidis.Client) {
	for {
		key := randString(10)
		value := randString(10)

		err := client.Do(ctx, client.B().Setex().Key(fmt.Sprintf("test1:%s", key)).Seconds(120).Value(value).Build()).Error()
		if err != nil {
			println("ERROR:", err.Error())
			time.Sleep(time.Second)
		}
	}
}

func control(ctx context.Context, client rueidis.Client) {
	for {
		c, _ := client.Do(ctx, client.B().Dbsize().Build()).AsInt64()
		println("dbsize", c)
		time.Sleep(time.Second)
	}

}
func randString(n int) string {
	var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")

	b := make([]rune, n)
	for i := range b {
		b[i] = letters[rand.Intn(len(letters))]
	}

	return strings.TrimSpace(string(b))
}

The throughput was greater than 1M rpm and the cache CPU never exceeded 60%. I ran the tests from a c5.xlarge instance.

I haven't been able to run tests with the code you gave me or with version v1.0.0 yet, but I'll send you updates as soon as I have them (either today or tomorrow).

[rueian]

Hi @arielzach, I will try your code again as soon as possible. Also, note that rueidis v1.0.0 was released under redis, and you will need to change its import path.

[arielzach]

Hi @rueian

I've been testing your code and my code, and everything worked correctly with version v1.0.0. I tested adding/removing shards and failover, which had a small downtime, but that is expected.

Thank you very much for the help.

[rueian]

Hi @arielzach,

Glad to hear that. I have also tested your code with v1.0.0. It looked good on my side as well.