Skip to content

Conversation

@tanvipenumudy
Copy link
Contributor

What changes were proposed in this pull request?

We have introduced a lightweight listKeys API as part of HDDS-9079. Implement a lightweight listKeys API (PR #5115), leveraging a trimmed KeyInfo (BasicKeyInfo) proto object to exclude less relevant fields, such as KeyLocations, which have limited utility on the client side. This implementation utilizes newer server-side Ozone key encapsulations and proto objects as part of HDDS-9079.

message BasicKeyInfo {
    optional string keyName = 1;
    optional uint64 dataSize = 2;
    optional uint64 creationTime = 3;
    optional uint64 modificationTime = 4;
    optional hadoop.hdds.ReplicationType type = 5;
    optional hadoop.hdds.ReplicationFactor factor = 6;
    optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 7;
}

The primary goal of this enhancement is to skip the deserialization of metadata specific to block information, etc. thus reducing the performance overhead when listing keys. Additionally, we aim to further improve performance by selectively populating the below fields server-side only when the key's configuration differs from that of its associated bucket:

    optional uint64 modificationTime = 4;
    optional hadoop.hdds.ReplicationType type = 5;
    optional hadoop.hdds.ReplicationFactor factor = 6;
    optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 7;

This optimization should lead to reduced payload and server-side processing unless the key is replicated in a configuration distinct from the bucket's default settings. Addressing the review comment as part of this PR.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9373

How was this patch tested?

Existing test cases should cover the code changes

@duongkame
Copy link
Contributor

duongkame commented Oct 16, 2023

is it ready to be reviewed? @tanvipenumudy Looks like it's been in draft for a while.

@tanvipenumudy
Copy link
Contributor Author

@duongkame looks like there are a few CI/CD failures - will be taking a look at this, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants