Skip to content

HDDS-4468. Fix Goofys listBucket large than 1000 objects will stuck forever #1595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bharatviswa504 merged 5 commits into from
Nov 23, 2020
Merged

HDDS-4468. Fix Goofys listBucket large than 1000 objects will stuck forever #1595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b48ed3c
HDDS-4468. Fix Goofys listBucket large than 1000 objects will stuck f…
maobaolong Nov 16, 2020
2fa1b3a
HDDS-4468. Add necessary comments
maobaolong Nov 17, 2020
5c77886
HDDS-4468. Update comments.
maobaolong Nov 18, 2020
0e57920
Address comments.
maobaolong Nov 19, 2020
008c372
set marker to response also when not null
maobaolong Nov 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,10 @@ public Response list(
ContinueToken decodedToken =
ContinueToken.decodeFromString(continueToken);

// Assign marker to startAfter. for the compatibility of aws api v1
if (startAfter == null && marker != null) {
startAfter = marker;
}
if (startAfter != null && continueToken != null) {
// If continuation token and start after both are provided, then we
// ignore start After
Expand All @@ -129,7 +133,7 @@ public Response list(
response.setDelimiter(delimiter);
response.setName(bucketName);
response.setPrefix(prefix);
response.setMarker("");
response.setMarker(marker == null ? "" : marker);
response.setMaxKeys(maxKeys);
response.setEncodingType(ENCODING_TYPE);
response.setTruncated(false);
Expand Down Expand Up @@ -187,6 +191,8 @@ public Response list(
response.setTruncated(true);
ContinueToken nextToken = new ContinueToken(lastKey, prevDir);
response.setNextToken(nextToken.encodeToString());
// Set nextMarker to be lastKey. for the compatibility of aws api v1
response.setNextMarker(lastKey);
} else {
response.setTruncated(false);
}
Expand Down
11 changes: 11 additions & 0 deletions ...ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ListObjectResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ public class ListObjectResponse {
@XmlElement(name = "NextContinuationToken")
private String nextToken;

@XmlElement(name = "NextMarker")
private String nextMarker;

@XmlElement(name = "continueToken")
private String continueToken;

Expand Down Expand Up @@ -177,4 +180,12 @@ public int getKeyCount() {
public void setKeyCount(int keyCount) {
this.keyCount = keyCount;
}

public void setNextMarker(String nextMarker) {
this.nextMarker = nextMarker;
}

public String getNextMarker() {
return nextMarker;
}
}