[CORE-8848] rptest: adjust compaction settings in datalake/compaction_test

[WillemKauf]

Because we now schedule adjacent segment compaction after sliding window compaction, this test was having trouble trying to reach the desired number of segments while producing.

Increase the timeout as well as the log_compaction_interval_ms to allow the test to reach the desired number of segments.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

• none

[vbotbuildovich]

Retry command for Build#61801

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/compaction_test.py::CompactionGapsTest.test_translation_no_gaps@{"catalog_type":"rest_jdbc","cloud_storage_type":1}
tests/rptest/tests/datalake/compaction_test.py::CompactionGapsTest.test_translation_no_gaps@{"catalog_type":"rest_hadoop","cloud_storage_type":1}

[vbotbuildovich]

CI test results

test results on build#61801

test_id	test_kind	job_url	test_status	passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f654-e8be-45a4-bed8-ee01e2d5360c	FLAKY	1/3
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f659-106a-4604-aea7-d4bd3226eda9	FLAKY	1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f654-e8bf-4294-9f32-b1dda3a2200c	FLAKY	1/2
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f659-106a-4604-aea7-d4bd3226eda9	FAIL	0/20
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f659-1067-43af-bea3-8ac5c24324ae	FAIL	0/20
rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f654-e8bf-4294-9f32-b1dda3a2200c	FLAKY	1/2
rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f659-106a-4604-aea7-d4bd3226eda9	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f659-1069-48f3-84cb-b3b611125f0e	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f659-106a-4604-aea7-d4bd3226eda9	FLAKY	1/2
rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_consumer_interruption	ducktape	https://buildkite.com/redpanda/redpanda/builds/61801#0194f654-e8bc-47a1-adbf-12c11d80ba67	FLAKY	1/2

test results on build#61807

test_id	test_kind	job_url	test_status	passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery	ducktape	https://buildkite.com/redpanda/redpanda/builds/61807#0194f77b-dd54-4034-90ce-1dcaaf816bcb	FLAKY	1/3
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade	ducktape	https://buildkite.com/redpanda/redpanda/builds/61807#0194f769-f0ba-4a14-8fdd-cadb456ac157	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61807#0194f769-f0bb-4508-8f27-faf0ea4b5bba	FLAKY	1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61807#0194f769-f0bd-44b4-a94b-ba347f321834	FLAKY	1/3
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic	ducktape	https://buildkite.com/redpanda/redpanda/builds/61807#0194f769-f0bd-44b4-a94b-ba347f321834	FLAKY	1/2
storage_single_thread_rpunit.storage_single_thread_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61807#0194f71f-d9d2-431e-a157-d1fde5eb99e2	FLAKY	1/2

test results on build#61813

test_id	test_kind	job_url	test_status	passed
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC	ducktape	https://buildkite.com/redpanda/redpanda/builds/61813#0194f854-3264-4611-a8d4-835f5f966905	FLAKY	30/34

[bharathv]

/ci-repeat 1
skip-build
skip-units
dt-repeat=30
tests/rptest/tests/datalake/compaction_test.py

[WillemKauf]

@bharathv the timeout issues in CompactionGapsTest seem fixed by this PR, but there are still some failures such as

ducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-21: Command 'python3 /tmp/compute_storage.py --data-dir=/var/lib/redpanda/data' returned non-zero exit status 1. Remote error message: b'Traceback (most recent call last):\n  File "/tmp/compute_storage.py", line 196, in <module>\n    output = compute_size(data_dir, args.sizes, args.md5, args.print_flat)\n  File "/tmp/compute_storage.py", line 151, in compute_size\n    for partition in safe_listdir(topic):\n  File "/tmp/compute_storage.py", line 89, in safe_listdir\n    return [f for f in p.iterdir()]\n  File "/tmp/compute_storage.py", line 89, in <listcomp>\n    return [f for f in p.iterdir()]\n  File "/usr/lib/python3.10/pathlib.py", line 1017, in iterdir\n    for name in self._accessor.listdir(self):\nNotADirectoryError: [Errno 20] Not a directory: \'/var/lib/redpanda/data/datalake_staging/0-2c771258-a4b6-4750-870b-238fa1da198f.parquet\'\n'

Separate issue?

[WillemKauf]

I think compute_storage.py should look like

try:                                                                                                                  
  return [f for f in p.iterdir()]                                                                                   
except FileNotFoundError:                                                                                             
   return []                                                                                                         
except NotADirectoryError:                                                                                            
  return []  

but I'm not sure yet, quite tired