HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state by mehakmeet · Pull Request #4331 · apache/hadoop

mehakmeet · 2022-05-19T06:28:03Z

ABFS rename fails intermittently when the Storage-blob tracking
metadata is in an incomplete state. This surfaces as the error code
404 and an error message of "RenameDestinationParentPathNotFound"

To mitigate this issue, when a request fails with this response.
the ABFS client issues a HEAD call on the source file
and then retry the rename op.

ABFS filesystem statistics track when this occurs with new counters
rename_recovery
metadata_incomplete_rename_failures
rename_path_attempts

This is very rare occurrence and appears to be triggered under certain
heavy load conditions, just as HADOOP-18163 is.

Contributed by Mehakmeet Singh.

…lete state

mehakmeet · 2022-05-19T06:30:00Z

CC: @mukund-thakur @steveloughran
Reproducing this exact issue seems tricky. Also, need to check if this is the only error code that corresponds to this issue, if the tracking metadata is in an incomplete state, can we not see other 404s too?

mehakmeet · 2022-05-23T12:06:48Z

Also, need to check if this is the only error code that corresponds to this issue, if the tracking metadata is in an incomplete state, can we not see other 404s too?

@snvijaya please provide some info on this if you could. Thanks.

steveloughran

made some suggestions. to really test that resilience, you will need to use mockito to inject a failure

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

steveloughran · 2022-05-24T21:56:40Z

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

  }

+  @Test
+  public void testRenameWithNoDestinationParentDir() throws Exception {


add similar test case for resilient rename api, here or in ITestAbfsManifestStoreOperations

snvijaya

Please have a look at the code comment given in AbfsClient.
As mentioned by Steve, driver test code uses mockito to recreate negative test cases.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

mehakmeet · 2022-06-06T05:16:21Z

Thanks for the review @snvijaya, had some pending changes, going to address your comments in the next commit alongside the change required to the condition of httpStatus code in the metadata incomplete scenario.

steveloughran

reviewed. the less we use mockito, the less to break. so use new() over mocking where possible

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientResult.java

...ls/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsRenameRetryRecovery.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

steveloughran

+1

Happy with is. Mehakmeet, is this ready to go in?

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientResult.java

mukund-thakur

few minor feedbacks.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsStatistic.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

...-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsRenameRetryRecovery.java

mehakmeet · 2022-06-18T02:57:32Z

Mehakmeet, is this ready to go in?

@steveloughran
Waiting on MSFT to provide a key detail on what kind of HttpStatus codes we can expect apart from ParentNotFound. Yet to hear from them. Apart from that this is ready to go in.

steveloughran

LGTM. +1

how about we merge this and see if it is enough to eliminate the problem -if we see different failures with the same fix, we can extend it.

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

steveloughran

was about to merge, did a final review, and realised that we should pick up the source etag and use it in the inner rename call

steveloughran · 2022-06-24T15:19:26Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

+
+          // Doing a HEAD call resolves the incomplete metadata state and
+          // then we can retry the rename operation.
+          getPathStatus(source, false, tracingContext);


I've had one more thought here. the path status contains the etag, doesn't it? so if sourceEtag was null, now we can set it. That way if the rename failure is followed
immediately buy the rename-failure-but-it-really-happened event of HADOOP-18163, we are lined up for recovery

steveloughran · 2022-06-24T15:22:09Z

really close to merge, just had one final thought before we hit the merge button. already updated the PR text in preparation for the merge (i was writing it up while looking at the code)

hadoop-yetus · 2022-06-27T11:43:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 18s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 50s		trunk passed
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 40s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 26s		trunk passed
+1 💚	shadedclient	23m 53s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	24m 16s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 23s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 11s		the patch passed
+1 💚	shadedclient	23m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 2s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		102m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/artifact/out/Dockerfile
GITHUB PR	#4331
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c50fd83515c3 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fc45b7a`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/testReport/
Max. process+thread count	533 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4331/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-06-27T15:35:04Z

due diligence question: what did you test against?

steveloughran · 2022-06-27T18:05:12Z

+1, merging.

tested myself against azure cardiff; transient failure of TestAbfsClientThrottlingAnalyzer in the parallel run; went away on a standalone one. that test is too brittle.

…omplete state (apache#4331) ABFS rename fails intermittently when the Storage-blob tracking metadata is in an incomplete state. This surfaces as the error code 404 and an error message of "RenameDestinationParentPathNotFound" To mitigate this issue, when a request fails with this response. the ABFS client issues a HEAD call on the source file and then retries the rename operation again ABFS filesystem statistics track when this occurs with new counters rename_recovery metadata_incomplete_rename_failures rename_path_attempts This is very rare occurrence and appears to be triggered under certain heavy load conditions, just as with HADOOP-18163. Contributed by Mehakmeet Singh.

HADOOP-18242. ABFS Rename Failure when tracking metadata is in incomp…

0afabed

…lete state

steveloughran requested changes May 24, 2022

View reviewed changes

snvijaya suggested changes Jun 6, 2022

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java Outdated Show resolved Hide resolved

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java Outdated Show resolved Hide resolved

HADOOP-18242. review comments

0f762d5

steveloughran requested changes Jun 9, 2022

View reviewed changes

HADOOP-18242. review comments + test fixes

7d7ec6f

steveloughran approved these changes Jun 17, 2022

View reviewed changes

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientResult.java Outdated Show resolved Hide resolved

mukund-thakur reviewed Jun 17, 2022

View reviewed changes

HADOOP-18242. review comments

9c85a9b

steveloughran reviewed Jun 20, 2022

View reviewed changes

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java Outdated Show resolved Hide resolved

mehakmeet added 2 commits June 22, 2022 12:27

HADOOP-18242. import reordering

39592f6

HADOOP-18242. javadocs

1b2672c

steveloughran requested changes Jun 24, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Jun 24, 2022

HADOOP-18242. Source etag extraction for future rename recovery.

fc45b7a

steveloughran merged commit 823f5ee into apache:trunk Jun 27, 2022

Comments

Conversation

mehakmeet commented May 19, 2022 • edited by steveloughran Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mehakmeet commented May 19, 2022

Uh oh!

mehakmeet commented May 23, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran May 24, 2022

Choose a reason for hiding this comment

Uh oh!

snvijaya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mehakmeet commented Jun 6, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mukund-thakur left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mehakmeet commented Jun 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Jun 24, 2022

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Jun 24, 2022

Uh oh!

hadoop-yetus commented Jun 27, 2022

Uh oh!

steveloughran commented Jun 27, 2022

Uh oh!

steveloughran commented Jun 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mehakmeet commented May 19, 2022 •

edited by steveloughran

Loading

mehakmeet commented Jun 18, 2022 •

edited

Loading