KAFKA-9548 Added SPIs and public classes/interfaces introduced in KIP-405 for tiered storage feature in Kafka. by satishd · Pull Request #10173 · apache/kafka

satishd · 2021-02-22T14:28:39Z

KAFKA-9548 Added SPIs and public classes/interfaces introduced in KIP-405 for tiered storage feature in Kafka.

KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol.

We introduced a few SPIs for plugging in log/index store and remote log metadata store.

This involves two parts

Storing the actual data in remote storage like HDFS, S3, or other cloud stroages.
Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic.

You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

satishd · 2021-02-23T18:09:20Z

@junrao @kowshik

junrao

@satishd : Thanks for the PR. A few comments below.

…-405 for tiered storage feature in Kafka. KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud stroages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage

satishd

thanks @junrao for the review. Addressed them in the latest commit.

junrao

@satishd : Thanks for the updated PR. A couple of more comments below.

satishd · 2021-03-03T05:27:45Z

@junrao thanks for the comment, added all public classes to javadoc section in build.gradle.

* Added new classes for java doc generation. * Fixed a few javadoc warnings in the newly added classes..

junrao

@satishd : Thanks for the updated PR. LGTM

kowshik

@satishd Thanks for the PR! LGTM. Just small comments. Feel free to address them later since this is merged.

satishd · 2021-03-30T07:50:31Z

@kowshik @ijuma thanks for your review. Raised #10424 to address your comments.

Replaced File with Path in LogSegment Data. This is a followup of #10173 Reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com>

Replaced File with Path in LogSegment Data. This is a followup of apache#10173 Reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com>

…-405 for tiered storage feature in Kafka. (apache#10173) KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage Reviewers: Jun Rao <junrao@gmail.com>

Replaced File with Path in LogSegment Data. This is a followup of apache#10173 Reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com>

…-405 for tiered storage feature in Kafka. (apache#10173) KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage Reviewers: Jun Rao <junrao@gmail.com>

Replaced File with Path in LogSegment Data. This is a followup of apache#10173 Reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com>

…-405 for tiered storage feature in Kafka. (apache#10173) Summary: KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage apache-reviewers: Jun Rao <junrao@gmail.com> (cherry picked from commit 0d9a95a) Reviewers: #ldap_kafka_admins, kchandraprakash Reviewed By: #ldap_kafka_admins, kchandraprakash JIRA Issues: DKAFC-868 Differential Revision: https://code.uberinternal.com/D6303151

Summary: Replaced File with Path in LogSegment Data. This is a followup of apache#10173 apache-reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com> (cherry picked from commit 3829df1) Reviewers: #ldap_kafka_admins, kchandraprakash Reviewed By: #ldap_kafka_admins, kchandraprakash JIRA Issues: DKAFC-868 Differential Revision: https://code.uberinternal.com/D6303171

…-405 for tiered storage feature in Kafka. (apache#10173) Summary: KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage apache-reviewers: Jun Rao <junrao@gmail.com> (cherry picked from commit 0d9a95a) Reviewers: #ldap_kafka_admins, kchandraprakash Reviewed By: #ldap_kafka_admins, kchandraprakash JIRA Issues: DKAFC-868 Differential Revision: https://code.uberinternal.com/D6303151

Summary: Replaced File with Path in LogSegment Data. This is a followup of apache#10173 apache-reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com> (cherry picked from commit 3829df1) Reviewers: #ldap_kafka_admins, kchandraprakash Reviewed By: #ldap_kafka_admins, kchandraprakash JIRA Issues: DKAFC-868 Differential Revision: https://code.uberinternal.com/D6303171

satishd force-pushed the KAFKA-9548 branch 5 times, most recently from cf3c406 to afdc0a0 Compare March 2, 2021 01:09

junrao reviewed Mar 2, 2021

View reviewed changes

satishd force-pushed the KAFKA-9548 branch from afdc0a0 to 687908c Compare March 2, 2021 05:06

satishd added 2 commits March 2, 2021 11:05

Addressed review comments.

0f444b4

satishd commented Mar 2, 2021

View reviewed changes

satishd force-pushed the KAFKA-9548 branch from 687908c to 0f444b4 Compare March 2, 2021 07:46

junrao reviewed Mar 2, 2021

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogMetadataManager.java

Comment thread clients/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java

Added new classes for java doc generation.

c0c79ed

* Added new classes for java doc generation. * Fixed a few javadoc warnings in the newly added classes..

satishd force-pushed the KAFKA-9548 branch from f4dc796 to c0c79ed Compare March 3, 2021 15:10

satishd commented Mar 3, 2021

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogMetadataManager.java

junrao approved these changes Mar 3, 2021

View reviewed changes

junrao merged commit 0d9a95a into apache:trunk Mar 3, 2021

ijuma reviewed Mar 9, 2021

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/server/log/remote/storage/LogSegmentData.java

kowshik reviewed Mar 10, 2021

View reviewed changes

satishd mentioned this pull request Mar 29, 2021

MINOR Replaced File with Path in LogSegmentData. #10424

Merged

3 tasks

junrao pushed a commit that referenced this pull request Mar 31, 2021

MINOR Replaced File with Path in LogSegmentData. (#10424)

3829df1

Replaced File with Path in LogSegment Data. This is a followup of #10173 Reviewers: Kowshik Prakasam <kprakasam@confluent.io>, Jun Rao <junrao@gmail.com>

Uh oh!

Conversation

satishd commented Feb 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Committer Checklist (excluded from commit message)

Uh oh!

satishd commented Feb 23, 2021

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

satishd left a comment

Choose a reason for hiding this comment

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

satishd commented Mar 3, 2021

Uh oh!

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kowshik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

satishd commented Mar 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

satishd commented Feb 22, 2021 •

edited

Loading