HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter #2729
Conversation
sumangala-patki
changed the title
| try { | ||
| op.execute(); | ||
| } catch (AzureBlobFileSystemException ex) { | ||
| String existingResource = |
There was a problem hiding this comment.
first check if the httpstatus code is 409 and (!isFile), then retrieve the existingResource header.
snvijaya
left a comment
snvijaya
left a comment
There was a problem hiding this comment.
One minor comment posted.
Recommend to change the Title and description as below:
Title: Toggle Store Mkdirs request overwrite parameter
Description:
In HDFS, mkdirs on an existing directory path is supposed to be a success response. To achieve this, the store backend call attempts mkdirs with overwrite=true. But on the backend, additional set properties operations also gets executed (such as LMT update) which is not a HDFS requirement and leads to unnecessary metadata update traffic.
In this PR, an option to have mkdirs executed with overwrite=false is introduced and is controlled over a config. Default is retained to overwrite=true until a related backend deployment is complete.
This PR also addresses a bug where mkdirs on an existing file path was returning success instead of throwing exception.
New config: fs.azure.enable.mkdir.overwrite [true by default]
sumangala-patki
changed the title
|
TEST RESULTS HNS Account Location: East US 2 |
…pache#2729) Contributed by Sumangala Patki. (cherry picked from commit fe633d4)
…pache#2729) Contributed by Sumangala Patki.
… parameter (apache#2729) (apache#2781) Contributed by Sumangala Patki. (cherry picked from commit fe633d4) Change-Id: Ibbd82f3cf2d28fa5438c9b1ee6b0e28217dc18df
5 participants
In HDFS, mkdirs on an existing directory path is supposed to be a success response. To achieve this, the store backend call attempts mkdirs with overwrite=true. But on the backend, additional set properties operations also gets executed (such as LMT update) which is not a HDFS requirement and leads to unnecessary metadata update traffic.
In this PR, an option to have mkdirs executed with overwrite=false is introduced and is controlled over a config. Default is retained to overwrite=true until a related backend deployment is complete.
This PR also addresses a bug where mkdirs on an existing file path was returning success instead of throwing exception.
New config:
fs.azure.enable.mkdir.overwrite[true by default]