HADOOP-17471. ABFS to collect IOStatistics #2731

mehakmeet · 2021-03-01T14:13:04Z

Tested by: mvn -T 16 -Dparallel-tests=abfs clean verify
Region: East US

[INFO] Results:
[INFO]
[INFO] Tests run: 95, Failures: 0, Errors: 0, Skipped: 0

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testCheckAccessForNonExistentFile »  Unexp...
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionALL:293->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionEXECUTE:218->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionNONE:204->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionREAD:233->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionREADEXECUTE:263->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionWRITE:248->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[ERROR]   ITestAzureBlobFileSystemCheckAccess.testFsActionWRITEEXECUTE:278->checkPrerequisites:306->setTestUserFs:88->checkIfConfigIsSet:319 » IllegalArgument
[INFO]
[ERROR] Tests run: 512, Failures: 0, Errors: 8, Skipped: 67

Illegal argument due to not using oauth config for these tests.

[INFO] Results:
[INFO]
[WARNING] Tests run: 256, Failures: 0, Errors: 0, Skipped: 77

mehakmeet · 2021-03-01T14:36:58Z

Need to remove DurationTrackers from streams, since now I am collecting them from AbfsRestOperation.

steveloughran · 2021-03-22T22:12:29Z

Looks great!
My main production suggestion is to make the counters a DurationTrackerFactory, giving us the option of feeding it in to anything we add in Hadoop common &c to take one.
test code: can you use assertThatStatisticMean() for your stats asserts -and if not, are changes needed?

mehakmeet · 2021-04-01T04:33:14Z

Extended AbfsCounters.java interface with DurationTrackerFactory.java, and converted the try-with-resources DurationTracking into

try {
      IOStatisticsBinding.trackDurationOfInvocation(abfsCounters,
          AbfsStatistic.getStatNameFromHttpCall(method),
          () -> completeExecute());
    } catch (IOException e) {
      throw new RuntimeException("Error while tracking Duration of an "
          + "AbfsRestOperation call", e);
    }

in the execute() method of AbfsRestOperation.java class, but I am facing this exception now:

java.lang.RuntimeException: Error while tracking Duration of an AbfsRestOperation call

	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:192)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:767)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:749)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:295)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:786)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:506)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.tryGetFileStatus(AzureBlobFileSystem.java:1035)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:135)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3460)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3565)
	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3518)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:592)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:604)
	at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.createFileSystem(AbstractAbfsIntegrationTest.java:271)
	at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: Operation failed: "The specified filesystem does not exist.", 404, HEAD, https://abfstorageacc.dfs.core.windows.net/abfs-testcontainer-72fd54ec-fe7b-42bc-a21c-61a175de9286/?upn=false&action=getAccessControl&timeout=90
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:225)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:190)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:454)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:188)
	... 29 more

Requests seem to return 404 with FileSystem not found exception. Tried to debug didn't get much info with that as well. Have you faced this issue before with DurationTrackingFactory @steveloughran ?

mehakmeet · 2021-04-12T10:38:20Z

My mistake. I was translating IOException into RuntimeException when it was expected to throw IOException while setup.

steveloughran

I'm submitting a review but I wrote it 10 days ago and forgot to hit "submit review"; possibly outdated now. sorry

...op-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsCounters.java

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsDurationTrackers.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsInputStreamStatistics.java

steveloughran · 2021-03-22T22:10:01Z

...adoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStreamStatistics.java

again, can we use assertThatStatisticMean()?

I am not able to add .isGreatherThan(<SOME_DOUBLE>) after assertThatStatisticMean(), I guess because this is an ObjectAssert<MeanStatistic> rather than DoubleAssert?

yes, I guess so. We'd probably need some conversion to get the mean/sum/count value. Let's not worry about that here

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsNetworkStatistics.java

… remove conflicts

hadoop-yetus · 2021-04-14T16:03:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 9s		trunk passed
+1 💚	compile	22m 38s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	19m 18s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	4m 1s		trunk passed
+1 💚	mvnsite	2m 9s		trunk passed
+1 💚	javadoc	1m 35s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 4s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 24s		trunk passed
+1 💚	shadedclient	16m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	21m 55s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	21m 55s		the patch passed
+1 💚	compile	19m 16s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	19m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 2s		the patch passed
+1 💚	mvnsite	2m 10s		the patch passed
+1 💚	javadoc	1m 33s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 3s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 47s		the patch passed
+1 💚	shadedclient	17m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 36s		hadoop-common in the patch passed.
+1 💚	unit	2m 2s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		208m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2731/5/artifact/out/Dockerfile
GITHUB PR	#2731
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 8b5358ce85a7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `16d1127`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2731/5/testReport/
Max. process+thread count	1274 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2731/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM; one change -to throwing an UncheckedIOException when wrapping an IOE.

+1 pending that change

steveloughran · 2021-04-22T15:15:43Z

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java

+    } catch (AzureBlobFileSystemException aze) {
+      throw aze;
+    } catch (IOException e) {
+      throw new RuntimeException("Error while tracking Duration of an "


change to UncheckedIOException; there's stuff in our functional package which can unwrap that down to an IOE as we always know the inner fault is a non-null IOE

…uration.

steveloughran · 2021-04-23T09:24:49Z

+1, merging. Thanks!

hadoop-yetus · 2021-04-23T09:47:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 22s		trunk passed
+1 💚	compile	21m 35s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	17m 58s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	3m 52s		trunk passed
+1 💚	mvnsite	2m 21s		trunk passed
+1 💚	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 16s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 31s		trunk passed
+1 💚	shadedclient	14m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 21s		the patch passed
+1 💚	compile	20m 4s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	20m 4s		the patch passed
+1 💚	compile	18m 5s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	18m 5s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 44s		the patch passed
+1 💚	mvnsite	2m 21s		the patch passed
+1 💚	javadoc	1m 47s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 16s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 54s		the patch passed
+1 💚	shadedclient	14m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 26s		hadoop-common in the patch passed.
+1 💚	unit	2m 12s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		197m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2731/6/artifact/out/Dockerfile
GITHUB PR	#2731
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 22858ad1a3f8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `665a0af`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2731/6/testReport/
Max. process+thread count	3158 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2731/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The ABFS Filesystem and its input and output streams now implement the IOStatisticSource interface and provide IOStatistics on their interactions with Azure Storage. This includes the min/max/mean durations of all REST API calls. Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>

The ABFS Filesystem and its input and output streams now implement the IOStatisticSource interface and provide IOStatistics on their interactions with Azure Storage. This includes the min/max/mean durations of all REST API calls. Note: This commit contains some changes from "CDPD-20510. HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark" in DurationTrackerFactory interface to keep AbfsCountersImpl class from breaking. Contributed by Mehakmeet Singh Conflicts: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java Change-Id: I07df5f558b00a7e548eda33dd6503f14337d5fdc

steveloughran reviewed Apr 12, 2021

View reviewed changes

Mehakmeet Singh added 5 commits April 14, 2021 17:00

HADOOP-17471. ABFS to collect IOStatistics

be517c6

HADOOP-17471. Remove DurationTrackers from ABFS streams and fix tests

8dc6c53

HADOOP-17471. Remove unused imports and javadoc fix

62d755f

HADOOP-17471. steve's review comments

a6506f7

HADOOP-17471. Add exception to method signature in a test + rebase to…

16d1127

… remove conflicts

mehakmeet force-pushed the HADOOP-17471 branch from 7a5913c to 16d1127 Compare April 14, 2021 12:33

mehakmeet requested a review from steveloughran April 20, 2021 06:45

steveloughran requested changes Apr 22, 2021

View reviewed changes

HADOOP-17471. UncheckedIOException when catching IOE while tracking d…

665a0af

…uration.

steveloughran merged commit 6085f09 into apache:trunk Apr 23, 2021

HADOOP-17471. ABFS to collect IOStatistics #2731

HADOOP-17471. ABFS to collect IOStatistics #2731

Uh oh!

Conversation

mehakmeet commented Mar 1, 2021

Uh oh!

mehakmeet commented Mar 1, 2021

Uh oh!

steveloughran commented Mar 22, 2021

Uh oh!

mehakmeet commented Apr 1, 2021

Uh oh!

mehakmeet commented Apr 12, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveloughran Mar 22, 2021

Choose a reason for hiding this comment

Uh oh!

mehakmeet Apr 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 22, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Apr 14, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 22, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Apr 23, 2021

Uh oh!

hadoop-yetus commented Apr 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mehakmeet Apr 12, 2021 •

edited

Loading