Unable to connect via TLS to DigitalOcean Managed Redis

[lylo]

I am experiencing issues connecting to a DigitalOcean Managed Redis instance and I wondered if someone might be able to help?

I can successfully connect from my server using redli:

redli -h myredis.ondigitalocean.com -p 25061 -a [PASSWORD] --tls
>ping
PONG

However when I configure my Caddy instance with the Redis module as follows I am seeing an error in the syslog:

  "storage": {
    "Client": null,
    "ClientLocker": null,
    "Logger": null,
    "address": "myredis.ondigitalocean.com:25061",
    "aes_key": "redistls-01234567890-caddytls-32",
    "db": 1,
    "key_prefix": "caddytls",
    "module": "redis",
    "password": "[PASSWORD]",
    "timeout": 5,
    "tls_enabled": true,
    "tls_insecure": false,
    "value_prefix": "caddy-storage-redis"
  },

The error I am seeing is:

Oct  4 15:05:17 caddy[2405]: {"level":"info","ts":1601823917.91949,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
Oct  4 15:05:17 caddy[2405]: {"level":"info","ts":1601823917.922724,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
Oct  4 15:05:17 caddy[2405]: {"level":"info","ts":1601823917.9232965,"logger":"caddy.storage.redis","msg":"TLS Storage are using Redis, on myredis.ondigitalocean.com:25061"}
Oct  4 15:05:17 caddy[2405]: run: loading initial config: loading new config: loading storage module: loading module 'redis': provision caddy.storage.redis: EOF
Oct  4 15:05:17 systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Oct  4 15:05:17 systemd[1]: caddy.service: Failed with result 'exit-code'.

Is anyone able to help me towards finding a solution at all? Many thanks.

[robgordon89]

hey @lylo im actually in the same exact state at the moment, I thought it was me. have you got anywhere with this since posting?

[lylo]

hey @lylo im actually in the same exact state at the moment, I thought it was me. have you got anywhere with this since posting?

Not yet, I only posted this today. I'm at a bit of a dead end tbh. I've tried looking at the code for this module but it's unclear to me how caddy-tlsredis is actually setting up TLS. The only line I can see is this:

caddy-tlsredis/storageredis.go

Line 358 in d368f4b

redisClient.Options().TLSConfig = &tls.Config{

[robgordon89]

what are you seeing in syslog, im running in k8s so debug is not great for me.

Unfortunately I have just seen it might be related to the redis client in use

redis/go-redis#1306

[lylo]

My syslog is in the bug report but I'll post here again for good measure:

Oct  4 15:05:17 caddy[2405]: {"level":"info","ts":1601823917.91949,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
Oct  4 15:05:17 caddy[2405]: {"level":"info","ts":1601823917.922724,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
Oct  4 15:05:17 caddy[2405]: {"level":"info","ts":1601823917.9232965,"logger":"caddy.storage.redis","msg":"TLS Storage are using Redis, on myredis.ondigitalocean.com:25061"}
Oct  4 15:05:17 caddy[2405]: run: loading initial config: loading new config: loading storage module: loading module 'redis': provision caddy.storage.redis: EOF
Oct  4 15:05:17 systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Oct  4 15:05:17 systemd[1]: caddy.service: Failed with result 'exit-code'.

[robgordon89]

hey sorry how did i miss that lol, im just running a quick test with go-redis to see if i can connect locally (without caddy etc)

[robgordon89]

ok resolved it, for some reason it doesn't take notice of the config in the CaddyFile, it works if you set the values via env's

Give it a go and let me known.

[lylo]

@robgordon89 this doesn't seem to work for me. I have defined the following in /etc/environment

CADDY_CLUSTERING_REDIS_HOST="myredis.ondigitalocean.com"
CADDY_CLUSTERING_REDIS_PORT="25061"
CADDY_CLUSTERING_REDIS_PASSWORD="[PASSWORD]"
CADDY_CLUSTERING_REDIS_DB=1
CADDY_CLUSTERING_REDIS_TIMEOUT=5
CADDY_CLUSTERING_REDIS_AESKEY="redistls-01234567890-caddytls-32"
CADDY_CLUSTERING_REDIS_KEYPREFIX="caddytls"
CADDY_CLUSTERING_REDIS_VALUEPREFIX="caddy-storage-redis"
CADDY_CLUSTERING_REDIS_TLS=true
CADDY_CLUSTERING_REDIS_TLS_INSECURE=false

I then restarted Caddy and I'm still saying the same error in the syslog.

Are you able to share a redacted Caddyfile and environment variables? Also are you on Caddy 2.2.0?

[robgordon89]

Not sure what your running on but is your environment file sourced ?

Im running on kubernetes and everything seems to be fine with envs it appears the config parser here is not picking up the tls options from the caddyfile

I have pretty much the same as you in envs, I also made the caddyfile read from the same envs just incase.

I'll pull out my conf for you in about 20/30 mins

[robgordon89]

Hey I am using k8s but please find details below, also check your firewall on your redis cluster.

here is the storage part of my config this is there just for me to keep ref.

storage redis {
    host        {$CADDY_CLUSTERING_REDIS_HOST}
    port        {$CADDY_CLUSTERING_REDIS_PORT}
    password    {$CADDY_CLUSTERING_REDIS_PASSWORD}
    db          {$CADDY_CLUSTERING_REDIS_DB}
    tls_enabled {$CADDY_CLUSTERING_REDIS_TLS}
    aes_key     {$CADDY_CLUSTERING_REDIS_AESKEY}
}

my envs are the same as yours currently, just one addition CADDY_CLUSTERING="redis"

I am on 2.2.0 and caddy-tlsredis v0.2.2

[lylo]

Thank you @robgordon89. I have it working now. Turns out that starting Caddy fails when I run it via systemctl for some reason. Running caddy start loads fine 🤦 Thanks for your help

[lylo]

So I think this issue stands in that hard-coding the host, port etc in the Caddy file doesn't appear to work whereas referencing the environment variables does.

[gamalan]

Thanks for noticing, it seems the environment value are overwriting or i misinterpreted the lifecycle.

The value is actually not hardcoded, but i set default value in case nothing is set in config or environment. If you look at func (rd *RedisStorage) GetConfigValue() L#256 it should only rewrite if config value are empty, and somehow at that point it become empty. Not sure why.

I push some fix that might be the issue on branch fix-bug-env-overwrite.

Busy at the moment. I check it again in next week. Sorry.

[regbo]

hey, I think this is the same thing happening in issue #13 I'll try the fix now.

[regbo]

Your fix didn't seem to work, but I think this PR does #16

[robgordon89]

Seems a lot cleaner also I will try and give this a try later today.

[gamalan]

okay, big thanks @regbo