Skip to content

HADOOP-19393. ABFS: Returning FileAlreadyExists Exception for UnauthorizedBlobOverwrite Rename Errors #7312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steveloughran merged 7 commits into from
Feb 17, 2025
+28 −0
Merged

HADOOP-19393. ABFS: Returning FileAlreadyExists Exception for UnauthorizedBlobOverwrite Rename Errors #7312

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
69228d3
throwing FileAlreadyExists exception for 403 rename errors
manika137 Jan 22, 2025
2b8b130
correct import sequence
manika137 Jan 22, 2025
00b426f
added test
manika137 Jan 22, 2025
401052c
minor corrections
manika137 Jan 23, 2025
001e137
resolving comments
manika137 Jan 24, 2025
6822054
checkstyle fix
manika137 Jan 26, 2025
0ec4803
Merge branch 'trunk' into renameErrorMapping
manika137 Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ...src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AzureServiceErrorCode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ public enum AzureServiceErrorCode {
ACCOUNT_REQUIRES_HTTPS("AccountRequiresHttps", HttpURLConnection.HTTP_BAD_REQUEST, null),
MD5_MISMATCH("Md5Mismatch", HttpURLConnection.HTTP_BAD_REQUEST,
"The MD5 value specified in the request did not match with the MD5 value calculated by the server."),
UNKNOWN(null, -1, null);
UNKNOWN(null, -1, null),
UNAUTHORIZED_BLOB_OVERWRITE("UnauthorizedBlobOverwrite", HttpURLConnection.HTTP_FORBIDDEN, "This request is not authorized to perform blob overwrites.");
Copy link
Contributor

@steveloughran steveloughran Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: split the line with the text on the line below

Copy link
Contributor Author

@manika137 manika137 Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken

Copy link
Contributor

@anujmodi2021 anujmodi2021 Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It would be better to move it above Unknown or add with other 403 errors

Copy link
Contributor Author

@manika137 manika137 Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken


private final String errorCode;
private final int httpStatusCode;
Expand Down
7 changes: 7 additions & 0 deletions ...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.ObjectMapper;

import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;
import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
Expand Down Expand Up @@ -127,6 +128,7 @@
import static org.apache.hadoop.fs.azurebfs.constants.HttpQueryParams.QUERY_PARAM_RETAIN_UNCOMMITTED_DATA;
import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND;
import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.UNAUTHORIZED_BLOB_OVERWRITE;

/**
* AbfsClient interacting with the DFS Endpoint.
Expand Down Expand Up @@ -613,6 +615,11 @@ public AbfsClientRenameResult renamePath(
throw e;
}

if(op.getResult().getStorageErrorCode()
.equals(UNAUTHORIZED_BLOB_OVERWRITE.getErrorCode())){
throw new FileAlreadyExistsException("File already exists." );
Copy link
Contributor

@snvijaya snvijaya Jan 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error string should be "File already exists or request not authorized for blob overrides", as this can happen when destination already exists or if SAS is missing "m".

Copy link
Contributor Author

@manika137 manika137 Jan 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Taken.

Copy link
Contributor Author

@manika137 manika137 Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, we're observing other 403 errors with mismatching SAS tokens/invalid permissions. We see UNAUTHORIZED_BLOB_OVERWRITE only with rename overwrites. Making changes accordingly.

}

// ref: HADOOP-18242. Rename failure occurring due to a rare case of
// tracking metadata being in incomplete state.
if (op.getResult().getStorageErrorCode()
Expand Down
Loading