[Enhancement]: Adding fetch from tiered storage system test

[showuon]

Related problem

No response

Suggested solution

In #10149, we added system test for tiered storage. But the test case testTieredStorageWithAivenPlugin only test the "upload" path, which is the local data can successfully upload log segments to the remote storage. We didn't have test for the "download" path, which is the consumer fetch data from remote storage.

I think we should add some more verification like:

1. Making sure the local log segments are deleted after uploading to remote storage. We have set

.addToConfig("file.delete.delay.ms", 1000)
.addToConfig("local.retention.ms", 1000)

It should be quick
2. Create a consumer to fetch from the beginning offset, it should read from the remote storage and get the expected record.
3. I'm not sure if we want to test the delete remote log path. If so, we can create a tiered storage enabled topic with short "retention.ms" or "retention.bytes", then verify it will delete the remote logs after expiration or size breached.

Alternatives

No response

Additional context

No response

[Frawless]

@showuon hey, current implementation of consumer uses from the begging config so it sounds to me that your suggested changes are already in place. Or do you suggest some additional check like - start consuming only when Kafka local storage is empty ?

Regarding 3. - is it something we should test or it's more likely plugin functionality?

[showuon]

Or do you suggest some additional check like - start consuming only when Kafka local storage is empty ?

Correct! That way, we can verify the path of fetching from remote storage works well.

[showuon]

Regarding 3. - is it something we should test or it's more likely plugin functionality?

We should test it. There are 2 parts of logic here:
(1) in the kafka side, we should be able to honor the retention setting (i.e. retention.ms/retention.bytes) to delete the expired log segments.
(2) In the remote storage, we will invoke the methods via plugin to delete the remote log segment.

So, yes, you are right, it's partially the plugin's functionality. But there is also part of Kafka's logic involved. Like we tested the log uploading to remote storage part, we want to verify the kafka side logic works well, and "assume" the Aiven's plugin works as expected.

Does that make sense?

[scholzj]

Discussed on the community call on 10.7.2024: This should be addressed. @Frawless will take a look.