Redis: delete and write data

[Dmitriy-Smol]

We want to use a DSL supporting EL to delete (DEL, SREM) and write data (SADD) to Redis, similar to a Redis feeder.
The motivation to use the Redis feeder this way is that we often use Redis to write, update and read OAuth tokens in scripts.
Would this violate the feeders concept?
It is planned to commit this?
And will you be ready to accept this implementation if we do it by adding redis command based on gatling-redis library?

Example DSL:

class RedisScenario {
 val redisPool = new RedisClientPool("localhost", 6379)
 def myRedisSADDFeeder(value: String, values: String*) = redisFeeder(redisPool, "key", value, values).SADD
 def myRedisDELFeeder(key: String, keys: String*) = redisFeeder(redisPool, key, keys).DEL
 def myRedisSREMFeeder(value: String, values: String*) = redisFeeder(redisPool, "key", value, values).SREM
  
 val getParameter: HttpRequestBuilder = http("GET PARAMETER")
  .get("/")
  .check(regex("""SPDX-License-Identifier: (\w.+)""").saveAs("value_1"))
  .check(regex("""Copyright (\w.+)""").saveAs("value_2"))

 val redisGetTokenScenario: ScenarioBuilder = scenario("Redis scenario")
  .exec(getParameter)
  .feed(myRedisSADDFeeder("${value_1}", "${value_2}"))
  .feed(myRedisDELFeeder("key", "key1"))
  .feed(myRedisSREMFeeder("${value_1}", "${value_2}"))

}

[slandelle]

Providing a feeder to read virtual user data from redis made perfect sense. It's the up to the user to store this data with his favorite redis client at the desired time, typically prior to launching the test.

Now, you're suggesting to have a way to also write operations.
IMHO:

• this is beyond the scope of Feeder, which is an alias for Iterator[Map[String, Any]]
• this actually feels like having a real redis protocol support, which would be something great to have!

WDYT? Is this something you would be interested in contributing?

[Dmitriy-Smol]

How do you see the implementation of Redis support from your side? Internal protocol or extension?

[slandelle]

Sorry, I don't understand your question. What do you mean by "internal protocol" vs "extension"?

[Dmitriy-Smol]

Internal protocol is the ability to call redis commands from the gatling package itself, without external dependencies. For example http protocol.

Extension examples are located on the page in the Third Parties section: gRPC plugin, Kafka Plugin, AMQP Plugin etc.

[slandelle]

OK, so you're referring to official plugins in the Gatling organization vs third party ones.

Third party plugins.

pros:

• you're the captain of your ship: you don't have to agree with anyone any any decision: design, release
• you own the project

cons:

• you're on your own
• we can't endorse nor advertise any more than list on the aforementioned page
• we can't help maintaining, in particular when changes occur on Gatling's internals
• at one point, you'll move to something else, stop maintaining and your project will be dead

Official plugins

pros:

• we participate in maintaining
• we help designing
• if you decide to move on, we'll keep on handling things
• more visibility and trust

cons:

• you don't have full control: you can't merge without a review from a Gatling employee and can't release yourself for security reasons
• we require that you donate the IP, we don't want to step onto any legal landmine

Your call

[Dmitriy-Smol]

What will be the requirements on your part when developing an official plugin?

Will it be necessary to implement in three languages: Scala, Java, Kotlin?

[slandelle]

What will be the requirements on your part when developing an official plugin?

A PR that contributes the code with a Gatling Corp copyright.

Will it be necessary to implement in three languages: Scala, Java, Kotlin?

Scala is a must. We can take care of the Java DSL as it can be a bit tricky. Regarding Kotlin, the goal for now is to have the Java DSL usage from Kotlin, not a distinct DSL with different paradigms.

[shoaib42]

Just wondering why this can't be achieved by a session function. (Been there done that, so it works, sorry I don't have the code with me)

1. Virtual user starts and adds a redis client to the session
2. Do you stuff
3. interact with redis add remove update in a session function

[ShadeCormac]

Hi,
Do you mind if i take this? Although i have very little experience with scala.

[slandelle]

@shoaib42 session function must be used to perform blocking calls, all the more network calls. This would hurt performance as the threads are shared.

[shoaib42]

@slandelle Yikes, I didn't think about that. I think what @Dmitriy-Smol has requested, is similar to the RPOPLPUSH that I did. Wouldn't it be a similar issue of blocking call with the feeder, if DEL SADD SREM were to be added? (Adding these would be pretty straight forward)

[slandelle]

Several things.

Using Feeder/feed for writing

That's a big no.

First, Feeder and feed are only meant for pulling records and copy them into Sessions. They are not meant to remove or modify data, all the more from an external system, and don't take any parameter. So something like requested doesn't belong there:

  .feed(myRedisSADDFeeder("${value_1}", "${value_2}"))
  .feed(myRedisDELFeeder("key", "key1"))
  .feed(myRedisSREMFeeder("${value_1}", "${value_2}"))

So now quick win by just add some more operations on RedisFeederBuilder.

Feeder currently only supports a blocking API

Feeder was designed to work with in-memory data: CSV and JSON files fully loaded in memory on boot and in-memory random data generator. Those are 100% fine with the blocking API.

Later, I also introduced the lazy feeders able to deal with humongous files. The files are on the local filesystem and the CSV parser is pretty fast, so it's mostly fine for most use cases.

Now, there's the case of the JDBC and REDIS feeders that perform a blocking network call on each Iterator.next(). I wouldn't use those for high throughput use cases.
In order for them to be efficient, we'd most likely need new implementations based on an async API, probably respectively on R2DBC and Lettuce. We'd probably have to move those into optional modules because of the extra dependencies.

exec(function) is a blocking API

It was designed to let users edit Session.

• It's sometimes diverted to push or pull data from a global shared in-memory store, which is fine.
• It's also sometimes diverted to write some data on disk, which can be an issue if your disk is slow or subject to hiccups.
• It's also sometimes diverted to perform blocking network IO and this is plain wrong. You would have to implement a proper Action.