Redis client-side caching connection deadlocks when health_check_interval is non-zero

[michael-chaos]

Version: 5.2.1

Platform: Windows 11 WSL -- Ubuntu 22.04.5 LTS (kernel 5.15.167.4-microsoft-standard-WSL2 SMP)

Description: Deadlock encountered when performing a get() call on a Redis connection with client-side caching enabled and the health_check_interval initializer arg non-zero.

        redis = redis.Redis(
            host=<host>,
            port=<port>,
            db=<db>,
            encoding='utf-8',
            protocol=3,  # Needing RESP3 protocol for client-side caching -- it is also supposed to be more performant
            cache_config=redis.cache.CacheConfig()  # Use client-side caching for performance
            decode_responses=True,
            retry_on_error=True,
            retry=redis.retry.Retry(redis.backoff.ExponentialBackoff(), 3),
            health_check_interval=1,
        )

Stack Trace:

_on_invalidation_callback (connection.py:929)		<<< _on_invalidation_callback() performs `with self._cache_lock`
handle_push_response (resp3.py:136)
_read_response (resp3.py:116)
read_response (resp3.py:28)
read_response (connection.py:588)
_send_ping (connection.py:511)
call_with_retry (retry.py:62)
check_health (connection.py:521)
send_packed_command (connection.py:529)
send_command (connection.py:556)
send_command (connection.py:819)			<<< send_command() performs `with self._cache_lock`
_send_command_parse_response (client.py:541)
<lambda> (client.py:568)
call_with_retry (retry.py:62)
_execute_command (client.py:567)
execute_command (client.py:559)
set (core.py:2335)

With my limited understanding of the Redis server and client implementations, it appears that the connection is processing a "PONG" response from the server in addition to a cache invalidation response/notification (i.e., some kind of multi-part response).

Within the connection.CacheProxyConnection class, self._cache_lock is a non-reentrant threading.Lock object. I suspect that changing it to a reentrant threading.RLock object may resolve this issue?