Skip to content

[FLUSS-2262][lake] Improved Stability for Iceberg Log Table Compaction Test #2265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rionmonster wants to merge 109 commits into from
Closed

[FLUSS-2262][lake] Improved Stability for Iceberg Log Table Compaction Test #2265

Changes from 1 commit
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
9a6b24b
[FLUSS-2262][lake] Improved Stability for Iceberg Log Table Compactio…
rionmonster Dec 27, 2025
8159dee
[FLUSS-2262][lake][server] Address lake table tiering stability durin…
rionmonster Jan 10, 2026
d901546
[FLUSS-2262][lake][server] Ensure pending tiering tables are removed …
rionmonster Jan 10, 2026
4447287
[lake/iceberg] Removal and clean-up for previous approaches
rionmonster Jan 28, 2026
b7a93c1
[lake/iceberg] Continued clean-up from previous iterations
rionmonster Jan 28, 2026
d980235
[common] switch to at.yawk.lz4:lz4-java due to CVE-2025-12183 (#2136)
sd4324530 Dec 28, 2025
ac06ca2
[lake] Support ADD COLUMN AT LAST for dataLake enabled tables (#2189)
buvb Dec 28, 2025
36e2a6f
[lake/paimon] Support NestedRow types for tiering paimon (#2260)
XuQianJin-Stars Dec 28, 2025
7cc346c
[docs] Use the "docker compose ps" to show the running containers (#2…
zuston Dec 28, 2025
cc18333
[server] Introduce new rebalance API (#1380)
swuferhong Dec 28, 2025
1d3b907
[Docs] Document the TypedAPI #2230 (#2261)
Prajwal-banakar Dec 28, 2025
aad81f3
[flink] Add RuntimeContext adapter for Flink 2.x compatibility (#2241)
sd4324530 Dec 29, 2025
224d5ff
[server] Support AddServerTag and RemoveServerTag (#1400)
swuferhong Dec 29, 2025
d886234
[client] Change SchemaNotExistException as retriable exception. (#2193)
loserwang1024 Dec 29, 2025
26eb8fb
[hotfix] Use CatalogTableAdapter rather than CatalogTable.newBuilder …
loserwang1024 Dec 29, 2025
c1a9bc6
[flink] Flink Source need to check tableId on recovery in case that t…
loserwang1024 Dec 29, 2025
dc90653
[common] Fix IndexOutOfBoundsException in ArrowArrayWriter when eleme…
Dec 12, 2025
2d0d9b0
[common Improve ArrowArrayWriter performance by replacing hardcoded h…
platinumhamburg Dec 31, 2025
1ed2c58
[lake] Record a file path storing log offsets in lake snapshot proper…
luoyuxia Dec 31, 2025
716fbb7
[kv] Implement basic Aggregate Merge Engine (#2255)
platinumhamburg Dec 31, 2025
8115d13
[hotfix]: Build expected results with partitions value, not hard code…
SML0127 Jan 4, 2026
daec805
[docs] update datatypes & paimon datatype mapping document (#2283)
Jackeyzhe Jan 4, 2026
09c1cf4
[spark] support spark batch write (#2277)
YannByron Jan 4, 2026
5b864e4
[common] Introduce MAP type for ARROW, COMPACTED and INDEXED formats …
XuQianJin-Stars Jan 5, 2026
e1e22e2
[hotfix] Fix compile problem of InternalRow#getMap() interface in Spa…
wuchong Jan 5, 2026
1782c6d
[hotfix] Fix spotless checkstyle problems
wuchong Jan 5, 2026
44a8c86
[lake/iceberg] Support tier array type for iceberg (#2266)
XuQianJin-Stars Jan 5, 2026
4096672
[hotfix] Fix compilation error: implement missing getMap method in Ic…
XuQianJin-Stars Jan 5, 2026
a099126
[kv] Support to report rocksdb metrics (#2282)
platinumhamburg Jan 5, 2026
5965ce8
[docs] Move 'Merge Engine' to the top-level of 'Table Design' (#2300)
Aditya41150 Jan 5, 2026
a3195d6
[spark] Add unit tests for spark SHOW TABLES command (#2304)
Yohahaha Jan 5, 2026
9d44aa3
[docs] Document COMPACTED table format (#2264)
Priyamanjare54 Jan 5, 2026
1b8f7e4
[server] Not permitted to enable datalake for tables created prior to…
LiebingYu Jan 7, 2026
badde92
[lake/paimon] Support map types for tiering paimon (#2308)
XuQianJin-Stars Jan 7, 2026
894daa9
[server] Fix ConcurrentModificationException when access getTable con…
LiebingYu Jan 7, 2026
cf653c8
[server] Servers with PERMANENT_OFFLINE tag are no longer assigned re…
LiebingYu Jan 7, 2026
293664c
[lake] fix potential dirty commit on re-create table (#2316)
zuston Jan 7, 2026
18f7bd1
[hotfix] use ltz_millis as paimon system column for timestamp dataty…
luoyuxia Jan 8, 2026
5a14061
[hotfix] Fix union read can't restore issue (#2321)
luoyuxia Jan 8, 2026
6472010
[server] Support generate and execute reblance plan (#1452)
swuferhong Jan 8, 2026
a36654f
[common] Add field_id for Nested Row. (#2322)
loserwang1024 Jan 8, 2026
85f0966
[flink] Refactor CALL procedure "sys.xxx_cluster_config" to support m…
sd4324530 Jan 8, 2026
e62373f
[hotfix] Fix the rebalance status cannot reach complete when the reba…
swuferhong Jan 8, 2026
7b5caf2
[docs] Add MAP type to data types and Paimon integration documentatio…
XuQianJin-Stars Jan 9, 2026
12ff308
[test] Add BucketingFunctionTest, this allows rust client side to als…
leekeiabstraction Jan 9, 2026
4f93ffa
[lake/iceberg] Support nested row types for Iceberg tiering (#2278)
SML0127 Jan 9, 2026
544cb4e
[hotfix] Add authentication for rebalance and controlled shutdown. (#…
LiebingYu Jan 9, 2026
b0a93fb
[docs]: updated data type mapping in iceberg (#2336)
SML0127 Jan 9, 2026
7600e8d
[NOTICE] Update copyright NOTICE year to 2026 (#2343)
wuchong Jan 11, 2026
e88e97e
[spark] Fix spark desc command output with partition info (#2313)
Yohahaha Jan 11, 2026
ef962be
[kv] include hidden potential internal flush time in KV flush latency…
zuston Jan 11, 2026
adbbd3d
[Task] Enhance tests for cluster rebalancing (#2315) (#2337)
Prajwal-banakar Jan 12, 2026
e4cb73c
[docs] Add alter table feature link in getting-started.md (#2348)
swuferhong Jan 12, 2026
1128946
[hotfix] Add authentication to the admin APIs that currently lack it …
swuferhong Jan 12, 2026
f45eabe
[doc] fix typo of 'table.datalake.enabled' in options.md (#2366)
zhaomin1423 Jan 14, 2026
44a1e16
[lake] Re-initial tiering splits for tables if reader failover happen…
beryllw Jan 15, 2026
6c1a155
[paimon] Add IT case for union read with evolved schema for Paimon (#…
ZuebeyirEser Jan 15, 2026
e3b094d
Rebalance procedure docs (#2355)
swuferhong Jan 15, 2026
6e15f53
[Flink] Support Partial Updates to the Flink Sink (#2042)
polyzos Jan 15, 2026
e3facdb
Change the ListRebalanceProcessProcedure result from String[] to row[…
sd4324530 Jan 16, 2026
d0a54b1
[spark] Add Spark show/add/drop partition support (#2314)
Yohahaha Jan 17, 2026
38ca1dd
[lake/lance] Refactor LanceArrowWriter to reuse fluss-common ArrowWri…
XuQianJin-Stars Jan 17, 2026
6e38036
[kv] Add basic implementation of AUTO_INCREMENT column (#2161)
beryllw Jan 6, 2026
2d86eed
[kv] Improve the implementation of AUTO_INCREMENT column
wuchong Jan 17, 2026
576c612
[server] Prevent remote log TTL expiration until tiering completes (#…
LiebingYu Jan 17, 2026
ed6040c
[spark] Add spark streaming write support (#2357)
Yohahaha Jan 18, 2026
513902c
Add RocksDB block cache configuration options for index and filter bl…
platinumhamburg Jan 18, 2026
9fc28b9
[test] Fix unstable test RebalanceITCase#testRebalanceForLogTable (#2…
swuferhong Jan 18, 2026
3075a41
[flink] Support PARTITION_DYNAMIC sink dynamic shuffle based on parti…
loserwang1024 Jan 15, 2026
d1464fd
[flink] Optimize sink statistics calculation to run only when necessary
wuchong Jan 17, 2026
c8d6f21
[flink] Fix the distribution mode support for FlussSink for DataStrea…
wuchong Jan 17, 2026
ed0f671
[tests] Cleanup ZK state in TableChangeWatcherTest to not let current…
vamossagar12 Jan 19, 2026
53f905a
[hotfix] Fix union read fail when from timestamp and projection enabl…
luoyuxia Jan 19, 2026
860e908
[docs] Add docs for rebalance (#2382)
swuferhong Jan 19, 2026
cff2a43
[lake] Support alter table.datalake.freshness (#2365)
zhaomin1423 Jan 20, 2026
e7309db
[flink] Flink integration with Aggregation Merge Engine (#2307)
platinumhamburg Jan 20, 2026
4a4670e
[kv] Add roaring bitmap aggregate function for aggregation merge engi…
platinumhamburg Jan 20, 2026
f4fdeaf
[client] move OffsetsInitializer to fluss-client (#2424)
YannByron Jan 21, 2026
071c572
[hotfix] Catalog related lake options should overwrite table lake rel…
luoyuxia Jan 21, 2026
f8e31dc
[test] Support manually trigger and wait for KV snapshots for FlussCl…
wuchong Jan 21, 2026
7035e24
[spark] support batch read from fluss cluster (#2377)
YannByron Jan 21, 2026
98e587f
[website] Update Fluss messaging (#2428)
polyzos Jan 22, 2026
540587a
[minor][flink] Remove unnecessary java notes (#2444)
leonardBang Jan 23, 2026
640485c
[docs] Add client feature support matrix documentation (#2445)
leekeiabstraction Jan 23, 2026
b10d912
[lake] report pending records even though no tiering is finished (#2453)
luoyuxia Jan 23, 2026
171875f
[hotfix][helm] Update image repository to use 'apache/fluss' instead …
morazow Jan 23, 2026
58bb4de
[coordinator] Refactor SchemaUpdate to delegate schema changes to Sch…
Prajwal-banakar Jan 23, 2026
fda1227
[flink] changelog read support for pk table without pushdown optimiza…
MehulBatra Jan 23, 2026
1a6e236
[lake] Make FlussLakeTiering pluggable to customize tiering job const…
luoyuxia Jan 24, 2026
eab70bb
[server] Fix FirstRowMergeEngine hang caused by empty log entries in …
loserwang1024 Jan 24, 2026
0909a08
[server] Update tablet server metadata cache when TableRegistration c…
LiebingYu Jan 24, 2026
d383c9f
[test] Fix failed testPkCompactedPollFromLatestNoRecords. (#2465)
SML0127 Jan 24, 2026
9a64fb5
[hotfix] Remove unstable "latest" scan startup mode test in Changelog…
wuchong Jan 25, 2026
5ab4f85
[flink] changelog read support for log table without pushdown optimiz…
MehulBatra Jan 25, 2026
f8eab26
[helm] Remove `tablet-server.id` from coordinator configuration optio…
morazow Jan 25, 2026
8a9a1e4
[test] Fix flaky AdjustIsrITCase#testIsrShrinkAndExpand by using acks…
ZuebeyirEser Jan 25, 2026
c711a5d
[doc/website] virtual tables documentation - $changelog for PK/Log Ta…
MehulBatra Jan 25, 2026
a9829e7
[test] Fix unstable test ReplicaTest.testRestore (#2469)
wuchong Jan 26, 2026
95feb64
[spark] Add Spark SQL ALTER table support for set/remove table proper…
Yohahaha Jan 26, 2026
09da97f
[hotfix] update messaging on github page (#2478)
polyzos Jan 26, 2026
a94c343
[build] Upgrade GitHub Actions for Node 24 compatibility (#2349)
sd4324530 Jan 26, 2026
5fd72ed
[lake] Tiering service support commit by time (#2185)
luoyuxia Jan 27, 2026
3f00b00
[kv] Add Producer Offset Snapshot for Exactly-Once semantics (#2434)
platinumhamburg Jan 27, 2026
10091bd
[hotfix] Fix unstable tiering source enumerator test (#2490)
luoyuxia Jan 27, 2026
2ad450f
[test] Add additional assertion to tests for TimestampLtz/Ntz to part…
leekeiabstraction Jan 27, 2026
90f5892
[fs/azure] Support Azure Blob Storage (#1941)
gkatzioura Jan 27, 2026
5c236d2
[flink] Fix Flink connector failure caused by unknown table options (…
LiebingYu Jan 28, 2026
95f4dd9
[spark] Add sparksql lake table catalog DDL suite for paimon (#2438)
Yohahaha Jan 28, 2026
51f516b
[s3] fix the size of fluss-fs-s3 is too big (#2419)
sd4324530 Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ...iceberg/src/test/java/org/apache/fluss/lake/iceberg/maintenance/IcebergRewriteITCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,9 @@ void testLogTableCompaction() throws Exception {
t1, t1Bucket, ++i, true, Collections.singletonList(row(1, "v1"))));
checkFileStatusInIcebergTable(t1, 3, false);

// Ensure tiering job has fully processed the previous writes
assertReplicaStatus(t1Bucket, i);

@luoyuxia luoyuxia Dec 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious about it. If the files count already is 3 in iceberg, shouldn't it also mean that the tiering has already tiered all records since we only write 3 records.

@rionmonster rionmonster Dec 29, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

I agree that it’s a bit odd. The additional replica assertion seems to alleviate any race conditions within this specific test case that we were seeing arise within the failing CI instances (at least based on my local testing to reproduce the issue).

It may simply be a testing artifact as opposed to a legitimate issue, but if it doesn’t seem like a fix, we can explore a few other avenues. It seemed like one of those common race conditions, so leveraging an existing function to help seemed like a decent approach. Happy to explore some additional avenues though, if we feel that we need a bit more exhaustive checks.

@luoyuxia luoyuxia Dec 29, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rionmonster I tried to reporduce it in my local. But fails. I think it'll be better to explore the root cause for I'm afraid of it is caused by another critical issue.

@rionmonster rionmonster Dec 29, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

Good to know! It seemed to be pretty consistent regarding pass/fail on my end, but clearly there’s something else at play.

I’ll do some more exploration and see what I find. Thanks for the feedback!

@rionmonster rionmonster Dec 30, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

I've been digging into this a bit further and it seems there's some disparity between the actual files being written and retrieving the latest offsets for those (specifically after any types of asynchronous operations, such as compaction, etc.). I think we may need some mechanism to improve the consistency, at least within the bounds of the tests.

I wrote a little monitor to run through several iterations of the tests to see what the state of various bits looked like during each iteration. Here's a sample of those:

Successful Test (Successfully Performs Compaction as Expected)
[MONITOR] log_table_33 - TableBucket{tableId=32, bucket=0}

IcebergFiles IcebergSnapshotId LakeSnapshotId ReplicaLakeSnapId ReplicaLakeLogEndOff Timestamp
0 -1 -1 -1 -1 1767113560800
0 -1 -1 -1 -1 1767113561046
... ... ... ... ... ...
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113615059
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113615307
2 2575057976625237982 2575057976625237982 5443797100773076340 4 1767113615557
2 2575057976625237982 2575057976625237982 5443797100773076340 4 1767113615808

Failing Test (File appeared to never be properly written before expected offset)
[MONITOR] log_table_34 - TableBucket{tableId=33, bucket=0}

IcebergFiles IcebergSnapshotId LakeSnapshotId ReplicaLakeSnapId ReplicaLakeLogEndOff Timestamp
0 -1 -1 -1 -1 1767113616327
... ... ... ... ... ...
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113861627
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113861882
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113862135
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113862381
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113862633 
[ASSERTION FAILURE] Expected offset 3 but got 2 for bucket TableBucket{tableId=33, bucket=0}
  Replica Lake Snapshot ID: 7273969972093574431
  Current State:
    Iceberg Files: 2
    Iceberg Snapshot ID: 7273969972093574431
    Lake Snapshot ID (from admin): 7273969972093574431
    Replica Lake Snapshot ID: 7273969972093574431
    Replica Lake Log End Offset: 2

I'm not sure if this is more of an artifact of the tests themselves or a legitimate issue. Any thoughts? Happy to continue digging. It feels like a race-condition due to the inconsistency, either on writing the files or reading stale offsets from the data lake directly. I'm sure in a real-world environment, this might be tolerable (as I suspect it's just a minor latency spike which would eventually resolve), but in the confines of a test, it's flaky.

@luoyuxia luoyuxia Jan 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was possible to request a table from the queue that was no longer actually in the "Pending" state.

@rionmonster
Could you please explain a bit about in which case the table in the queue that was no longer actually in the "Pending" state.? Although it's no in "Pending" state, the tiering service still can requst the table and do the tiering. Also, from the log, I can't see the log like "Fail to change state for table ".

To me, it looks like that the pendingTieringTables is empty, but I still can't figure out why

@luoyuxia luoyuxia Jan 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, could you please share me your reproduce branch so that I can reproduce it in my local env.

@rionmonster rionmonster Jan 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

Sure! Cleaning up some of my notes from the troubleshooting session, but hopefully this workflow makes sense:

  1. A table T becomes eligible and is assigned normally (Pending to Tiering).
  2. Later, a stale duplicate entry for T ends up in the pending queue (e.g., due to retries, late timer callbacks, or test churn).
  3. Before the fix, requestTable() would poll T from the pending queue and attempt to assign it again, even though its actual state was no longer Pending.
  4. That resulted in an invalid state transition attempt (non-Pending/Tiering to Tiering), but the table was still returned to the tiering service, causing things to become inconsistent between the pendingTieringTables and tieringStates.

The key difference with this change is that after it requestTable() actually verifies the table is in the Pending state before assigning it instead of just relying on its presence in the pending queue.

@luoyuxia luoyuxia Jan 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rionmonster

Later, a stale duplicate entry for T ends up in the pending queue (e.g., due to retries, late timer callbacks, or test churn).

As for the fail test, which case cause a stale duplicate T entering into pending queue? Do you have ever find any logs?

would poll T from the pending queue and attempt to assign it again, even though its actual state was no longer Pending

If it's a invalid state changte, we'll print

Fail to change state for table xxx

But I can't see such log in the fail ci

but the table was still returned to the tiering service, causing things to become inconsistent between the pendingTieringTables and tieringStates.

To me, it looks me like that the table won't be returned to the tiering service anymore from the log.

So, i'm a little doubt about this case.

Could you please share me your reproduce branch, I want reproduce it in my local env to find the root cause.

@rionmonster rionmonster Jan 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

Based on our private conversations exploring this, I'm including the following based on

Okay — I was able to reproduce the failure successfully with some extended debugging logs added throughout the LakeTableTieringManager. I've created a gist here with the logs. Reviewing over them, it looks like we have the following chain of events:

  • At 411079 we have a series of these "polled tableId=x but tablePath is null (state=null, epoch=null)" type of calls, which repeats 10+ times
  • After this burst we can see that the test fails (since rewrite/compaction didn't complete in time)

I think we can interpret this as:

  • Tiering service requested work (via requestTable())
  • Manager pulled table ids from pendingTieringTables but one of the following was true:
    • tablePaths didn't contain the id
    • tieringStates didn't contain the id
    • tableTierEpoch didn't contain the id
  • This means that the table ids were no longer registered (e.g., previous dropped/removed) but were still present in the pendingTieringTables
  • This causes any requestTable() calls to continually drain/recurse/loop over these stale entries which could delay processing of actual pending tables
  • This delay could be enough to cause the failed assertion

I think the proposed fixes would help with this, although we may need to consider adjusting the removeLakeTable() call as well to ensure we remove the requested table from the pending tables as well, which I'll add to the PR:

public void removeLakeTable(long tableId) {
    inLock(lock, () -> {
        // Omitted for brevity

        pendingTieringTables.remove(tableId);
    });
}

I've created a branch with these additional logs if you would like to explore it yourself at https://github.com/rionmonster/fluss/tree/for-yuxia-with-logs


// Write should trigger compaction now since the current data file count is greater or
// equal MIN_FILES_TO_COMPACT
flussRows.addAll(
Expand Down