[NEW] Introduce automated cross version and cross fork testing infrastructure

[hpatro]

DESCRIPTION

Introduce cross version/cross fork integration testing infrastructure. With the compatibility version release and planned new major version release, it will be good to improve the testing/release certification process. This will help Valkey to prepare for release(s) more confidently and avoid pain for user(s) during migration/upgrade(s).

Example Scenario:

Issue: redis/redis#12685

Redis 7.2 introduced cluster bus message extensions feature by default and it caused failure of engine upgrade from older version (i.e. Redis 6.2 or lower) due to message broadcasted from engine running 7.2 not being compatible in older versions.

PR to fix the issue during upgrade: #52

[hpatro]

@valkey-io/core-team WDYT ?

[zuiderkwast]

It would be great if anyone wants to add such tests, because we're a bit overloaded at this point. It doesn't have to be in TCL (maybe Python instead?) and it doesn't even have to be in this repo, if we're testing different forks. We can make a separate interop testing repo. It is about

• cluster bus (nodes of different versions in the same cluster)
• replication (primary/replica of different versions)
• RDB and AOF files (read and write by different versions)
• Anything else?

[pragnesh]

i am willing to spend time. if anyone guide me some starting point.

[madolson]

We can make a separate interop testing repo. It is about

I think it should be this repo, I don't think we want to start introducing tests elsewhere for now.

[zuiderkwast]

I'm fine either way, but what id'd like to see is some prototyping to get something running. Use any language, any client lib, but be ready to discard it later.

[zuiderkwast]

There are potential benefits of having a completely separate repo for this. It would need to check out various versions of valkey anyway (so it can't just run out of the checked out repo). It could also test valkey against various forks and versions. We can have test a cluster with mixed nodes of KeyDB, Dragonfly, Redict, Redis, Valkey, various versions. We can run redis testsuites on the valkey binary. Etc.

[suxb201]

@madolson @zuiderkwast @pragnesh @mattsta Hi everyone! Is anyone already working on this?
We currently have a Python test script that provides the same functionality as the TCL tests, but written in Python. It has been working well for the past two years and has significantly improved the efficiency of writing and debugging tests compared to TCL.
Here is a relatively old version: https://pypi.org/project/pybbt/
If you're interested, I'm willing to spend 1-2 weeks adapting it for Valkey and writing a few test examples.

[hpatro]

@suxb201 how is the test setup/teardown done? How long does a similar tcl test takes in python?

[suxb201]

@hpatro

1. To create an instance, use a clear statement.
2. Instances made inside an @subcase are safely destroyed when the @subcase ends.
3. Instances created within an @case will be destroyed when the @case concludes.
4. The @subcase is concurrent, which makes testing efficient.
5. There is only one @case per file, and the tests are composed of multiple files.
6. Logs, AOF and RDB files will be arranged in a temporary directory, making it easier to debug.

Here's a example:

from testsuite import *


# Master can replicate command longer than client-query-buffer-limit on replica
@subcase()
def replication_query_buffer_limit():
    t0 = Tair()  # create a process
    t1 = Tair()  # create another process
    t1.wait_slaveof(t0)  # call 'slaveof' and then wait for the replication connection to be set up
    t0.do("config", "set", "client-query-buffer-limit", 2000000)
    t1.do("config", "set", "client-query-buffer-limit", 1048576)  # 1024*1024 = 1mb

    value = "x" * 1100000
    ASSERT_TRUE(t0.do("set", "k", value))  # 2000000 > 1100000 > 1048576
    t0.wait_consistent()
    ASSERT_EQ(t1.do("get", "k"), value.encode())
    ASSERT_EQ(t1.digest(), t0.digest())


@case(tags=["flag0", "flag1", "flag2"])
def main():
    replication_query_buffer_limit()


if __name__ == '__main__':
    main()

[zuiderkwast]

It seems to me that porting the tests to a different language is a different topic. I think it can be discussed here: #94.

Personally, I think it would be beneficial to create a separate interoperability-testing repo where scripts and CI jobs can

• check out multiple versions of Valkey, Redis, clients and other tools
• test mixed clusters, e.g. test Pass extensions to node if extension processing is handled by it #52
• test replication between different versions
• create dumps (rdb and aof) and load them it in other versions, newer and older

@bjosv

[madolson]

Reference: https://github.com/tair-opensource/resp-compatibility. Alibaba has some compatibility testing already we should investigate.

[yangbodong22011]

I am the owner of resp-compatibility. It was originally designed to detect the Redis version compatible with the Redis-Like system from the interface, but it is also very suitable for version compatibility testing of Redis/Valkey itself. From the discussion in the issue, it can currently do

tests	YES or NO
api	YES
cluster bus (nodes of different versions in the same cluster)	NO(But can improve)
replication (primary/replica of different versions)	NO(But can improve)
RDB and AOF files	YES

We can:

• Run on daily-ci.
• Run this test when a new version is released.
• Run every time a PR is submitted (about 2 minutes).

Please share any thoughts you have, thank you.

[madolson]

@valkey-io/core-team We chatted a bit in the core meeting and want to have a vote if we want to pull in the compatibility test listed above and run it as part of the CI.

We still need a separate effort to do interversion operability tests (clusterbus and replication).

[madolson]

@zuiderkwast Did we secretly do this?

[zuiderkwast]

@zuiderkwast Did we secretly do this?

I must have pocket-clicked close...

[hwware]

@valkey-io/core-team We chatted a bit in the core meeting and want to have a vote if we want to pull in the compatibility test listed above and run it as part of the CI.

We still need a separate effort to do interversion operability tests (clusterbus and replication).

Agree, but Daily CI is not necessary.