Add scylladb

[mkorolyov]

ScyllaDB is an open-source distributed NoSQL wide-column data store. It gains wide popularity as a drop-in replacement for Cassandra but a significantly faster one.

This PR adds the ScyllaDB module.

[guy9]

Thanks @mkorolyov , interested in this as well

[mykaul]

I wish there was a way NOT to provide your own scylla.yaml, but merge your defaults with the default scylla.yaml - the end result is that you might be missing some new defaults if you bring your own scylla.yaml.

[mkorolyov]

I wish there was a way NOT to provide your own scylla.yaml, but merge your defaults with the default scylla.yaml - the end result is that you might be missing some new defaults if you bring your own scylla.yaml.

This looks like a separate feature for merging config files. Usually what I used to see in a majority of the applications is that you are using the default config file and overriding it with ENVs or flags or providing your own config file. Providing own config file which will still be merged with some default looks like implicit behavior which in my opinion will confuse users. WDYT?

[guy9]

Any updates on merging this?

[mykaul]

I wish there was a way NOT to provide your own scylla.yaml, but merge your defaults with the default scylla.yaml - the end result is that you might be missing some new defaults if you bring your own scylla.yaml.

This looks like a separate feature for merging config files. Usually what I used to see in a majority of the applications is that you are using the default config file and overriding it with ENVs or flags or providing your own config file. Providing own config file which will still be merged with some default looks like implicit behavior which in my opinion will confuse users. WDYT?

Yes, my ask is exactly this - let's use the default Scylla YAML that comes with Scylla, and allow overriding values in it.

[fruch]

those are not relevant to scylla, you need commandline option for skip_wait_for_gossip_to_settle

[fruch]

I think this needs to be checked, scylla docker doesn't use this variable

[vbazhmin]

Hello folks. Any updates on this one ? ;)

[eddumelendez]

Hi, I'll review the PR once the other comments have been addressed. It is great seeing Scylla team reviewing the PR.

[fruch]

Hi, I'll review the PR once the other comments have been addressed. It is great seeing Scylla team reviewing the PR.

hopefully it won't be dragged like it's counterpart:
testcontainers/testcontainers-python#167

[eddumelendez]

@fruch LMK once this is ready to review and will do it as soon as possible. I still see some comments that hasn't been resolved. So, that's the reason why I haven't reviewed yet.

[eddumelendez]

Hi, I know this is still in progress because I see some comments unresolved. But, JFTR it would be nice to add an integration test enabling ssl and add it to the docs.

[eddumelendez]

I'm updating the PR based on testcontainers/testcontainers-go#2919

[mkorolyov]

Hi @mykaul,

I've looked into implementing your suggestion, and it seems there are a few challenges with achieving this functionality:

1. ScyllaDB Configuration Dependency:
   The ScyllaDB service requires a single configuration file, typically located at /etc/scylla/scylla.yaml in the official Docker image. It's straightforward to replace this file in the container using a volume mount, e.g., -v ~/master_scylla.yaml:/etc/scylla/scylla.yaml.

2. Merging Custom Configuration:
   To merge a custom partial configuration with the default one, we would need to process and generate the final configuration file before the container starts and then mount it to the container.

3. Extracting Default Configuration:
   One potential approach is to extract the default configuration file directly from the Docker image without starting the container. This can be done using Docker commands, such as:

   docker create --name temp-container scylladb/scylla  
   docker cp temp-container:/etc/scylla/scylla.yaml ./scylla.yaml  
   docker rm temp-container  

While this method works, it feels like a bit of a hack to me, and I’m not entirely comfortable with introducing such a workaround into the workflow.

4. Maintaining Up-to-Date Configurations:
   Alternatively, the latest default ScyllaDB configuration file can be found in their repository here. However, storing a snapshot of this file in the testcontainers-java repository introduces a maintenance issue: it would become outdated as soon as updates are made upstream.

5. Dynamic Fetching Concerns:
   Dynamically fetching the latest configuration file from the ScyllaDB repository during container setup (e.g., via HTTP calls) is also problematic and may not align with project standards or be acceptable to the maintainers.

Given these constraints, do you have other suggestions for achieving this functionality? Specifically, how can we ensure the availability of an up-to-date configuration file while maintaining simplicity and compatibility with the existing workflow?

Looking forward to your thoughts!

[mkorolyov]

@eddumelendez Hey, thanks for stepping in and polishing PR. Just asking, why you removed custom config file related changes?

[mykaul]

@mkorolyov - thanks for looking into this. I think your point no. 1 makes a convincing argument - it's trivial to just pass a yaml file, it's quite standard, so anyone using the Scylla testcontainer can do it. Let's just document it.

[mkorolyov]

@mykaul Hey, added possibility to easily replace config file for test container and documented it.

@eddumelendez Hey, added ssl integration test and documented it.

Could you pls take a look so we could proceed with it?

Appreciate your help and time!

[eddumelendez]

@mkorolyov this test fails. Lacking ssl configuration? default image doesn't provide those env vars

[mkorolyov]

it was committed by mistake. cleaned that up, thanks!

[eddumelendez]

Thanks for adding this! I think cql cli command would not work unless is configured properly. We should configure it properly when ssl is enabled https://opensource.docs.scylladb.com/stable/operating-scylla/security/gen-cqlsh-file.html

[mkorolyov]

@eddumelendez yes, if ssl enabled on the server for client connections in order to cqlsh to work we need to configure ssl for it too, but it is not used here, in test containers, right? I didn't get your point, could you pls clarify what else you would like to see in this PR? thanks!

[eddumelendez]

Users can execute commands using via execInContainer method provided by Testcontainers API. Testcontainers should provide a proper configuration for the container to run properly. In this case, external clients will connect properly because the sdk client is configured but not inside the container. I think we should fix that.

[mkorolyov]

I've added a test example how to make cqlsh work with same ssl setup.