Skip to content

KAFKA-13603: Allow the empty active segment to have missing offset index during recovery #11345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
junrao merged 33 commits into from
Jan 27, 2022
Merged

KAFKA-13603: Allow the empty active segment to have missing offset index during recovery #11345

Changes from 4 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
ecb8692
Allow empty last segment to have missing offset index during recovery
ccding Sep 20, 2021
9904830
Revert "Allow empty last segment to have missing offset index during …
ccding Oct 1, 2021
ac6e9cd
Merge branch 'trunk' into last
ccding Oct 1, 2021
a2c51d9
flush empty active segments
ccding Oct 1, 2021
82913f0
add comment
ccding Oct 26, 2021
b915fe2
unit test
ccding Oct 27, 2021
4df64d8
address comments
ccding Nov 5, 2021
0f2f0a8
Merge branch 'trunk' into last
ccding Nov 8, 2021
c50bff2
flush at the right place
ccding Nov 17, 2021
603627f
Merge branch 'trunk' into last
ccding Nov 17, 2021
56d3f06
trigger test
ccding Nov 17, 2021
425a7a7
address comments
ccding Nov 19, 2021
f345831
Merge branch 'trunk' into last
ccding Nov 22, 2021
069ac5b
only do the inclusive flush in KRaft.close()
ccding Nov 22, 2021
049c856
Merge branch 'trunk' into last
ccding Nov 30, 2021
56d64cb
rename a bunch of variable/function names to address comments
ccding Nov 30, 2021
7bb37b8
fix typo and purpose
ccding Dec 1, 2021
40506eb
fix
ccding Dec 1, 2021
6f70af3
do not print error if the missing index file is after the recovery point
ccding Dec 7, 2021
23314ea
add log loader test
ccding Dec 20, 2021
d775f14
fix time stamp check
ccding Dec 20, 2021
4d88092
fix comment
ccding Dec 20, 2021
099cb42
fix log loader test
ccding Dec 21, 2021
229a537
Merge branch 'trunk' into last
ccding Dec 22, 2021
14239c0
trigger test
ccding Dec 22, 2021
8b7e0c9
Merge branch 'trunk' into last
ccding Dec 28, 2021
0c8e48b
Merge branch 'trunk' into last
ccding Jan 5, 2022
25821db
address comments
ccding Jan 10, 2022
47ba1d5
update comments and polish log messages
ccding Jan 11, 2022
885db15
improve log message
ccding Jan 24, 2022
5bb1a47
Merge branch 'trunk' into last
ccding Jan 27, 2022
e00906a
fix compile and improve log message
ccding Jan 27, 2022
bd21bb2
fix debug output
ccding Jan 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/scala/kafka/log/UnifiedLog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1505,7 +1505,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
/**
* Flush all local log segments
*/
def flush(): Unit = flush(logEndOffset)
def flush(): Unit = flush(logEndOffset + 1)

@kowshik kowshik Oct 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add unit test coverage for this change in UnifiedLogTest.scala?

@junrao junrao Oct 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a comment why we need to flush to logEndOffset + 1? Also, could we have a test that verifies the index file is present after an empty segment is rolled and the broker is shut down?

@ccding ccding Oct 1, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment. Will write it later. Was too busy yesterday and didn't have time to do so.


/**
* Flush local log segments for all offsets up to offset-1
Expand Down