Skip to content

Unify fallback dependent api #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anmolanmol1234 merged 14 commits into from
Jun 23, 2023
Merged

Unify fallback dependent api #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
ce92ee4
File status
anmolanmol1234 Jun 12, 2023
f03da1e
Make dependent fallback
anmolanmol1234 Jun 12, 2023
efc7b69
Test fixes
anmolanmol1234 Jun 12, 2023
e216b92
Test fix
anmolanmol1234 Jun 12, 2023
f0a0d8d
Removing comments
anmolanmol1234 Jun 13, 2023
cd0b502
Fix spacing
anmolanmol1234 Jun 13, 2023
91d3b12
Fix generic exception
anmolanmol1234 Jun 13, 2023
6db6bf3
Merge branch 'ABFS_3.3.2_dev' into unifyFallbackDependentAPI
anmolanmol1234 Jun 13, 2023
d6cce65
Resolving merge conflict
anmolanmol1234 Jun 20, 2023
6c8d08e
PR comments
anmolanmol1234 Jun 20, 2023
5ebd42c
Fix merge conflicts
anmolanmol1234 Jun 20, 2023
768b75a
Cast fix
anmolanmol1234 Jun 20, 2023
56c6914
Exception fix
anmolanmol1234 Jun 20, 2023
8766c54
space fix
anmolanmol1234 Jun 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions ...p-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
import org.apache.hadoop.fs.azurebfs.services.BlobProperty;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.azurebfs.services.OperativeEndpoint;
import org.apache.hadoop.fs.azurebfs.services.PathInformation;
import org.apache.hadoop.fs.azurebfs.services.PrefixMode;
import org.apache.hadoop.fs.azurebfs.services.RenameAtomicityUtils;
Expand Down Expand Up @@ -1002,25 +1003,23 @@ private FileStatus getFileStatus(final Path path,
statIncrement(CALL_GET_FILE_STATUS);
Path qualifiedPath = makeQualified(path);
FileStatus fileStatus;
PrefixMode prefixMode = getAbfsStore().getPrefixMode();
AbfsConfiguration abfsConfiguration = getAbfsStore().getAbfsConfiguration();

boolean useBlobEndpoint = !(OperativeEndpoint.isIngressEnabledOnDFS(prefixMode, abfsConfiguration) ||
Copy link
Collaborator

@snvijaya snvijaya Jun 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, let operativeEndpoint by the single point with the unified logic.

OperativeEndpoint.isMkdirEnabledOnDFS(prefixMode, abfsConfiguration) ||
OperativeEndpoint.isReadEnabledOnDFS(prefixMode, abfsConfiguration));
try {
Comment on lines +1009 to 1012
Copy link
Collaborator

@saxenapranav saxenapranav Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does redirect to dfs on mkdir/ingress/readEnabled matter here?

Copy link
Collaborator Author

@anmolanmol1234 anmolanmol1234 Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want the getFileStatus call to go to dfs in all cases where either of the configs is enabled, though this doesn't come as part of fallback but helps to maintain consistency in the logic

if (abfsStore.getPrefixMode() == PrefixMode.BLOB) {
/**
* Get File Status over Blob Endpoint will Have an additional call
* to check if directory is implicit.
*/
fileStatus = abfsStore.getFileStatusOverBlob(qualifiedPath,
tracingContext);
}
else {
fileStatus = abfsStore.getFileStatus(qualifiedPath,
tracingContext);
}
if (getAbfsStore().getAbfsConfiguration().getPrefixMode()
== PrefixMode.BLOB && fileStatus != null && fileStatus.isDirectory()
&&
abfsStore.isAtomicRenameKey(fileStatus.getPath().toUri().getPath()) &&
abfsStore.getRenamePendingFileStatusInDirectory(fileStatus,
tracingContext, useBlobEndpoint);
if (getAbfsStore().getPrefixMode() == PrefixMode.BLOB
&& fileStatus != null && fileStatus.isDirectory()
&& abfsStore.isAtomicRenameKey(fileStatus.getPath().toUri().getPath())
&& abfsStore.getRenamePendingFileStatusInDirectory(fileStatus,
tracingContext)) {
RenameAtomicityUtils renameAtomicityUtils = new RenameAtomicityUtils(
this,
Expand Down
Loading