Skip to content

Conversation

@anmolanmol1234
Copy link
Contributor

Removing write aggressiveness code and related metrics from trunk

@anmolanmol1234
Copy link
Contributor Author

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 892, Failures: 0, Errors: 0, Skipped: 223
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[ERROR] org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp.testUpdateDeepDirectoryStructureToRemote -- Time elapsed: 10.41 s <<< FAILURE!

[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemUserBoundSAS.testOAuthTokenProviderAndSASTokenFlow -- Time elapsed: 1.835 s <<< ERROR!
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemUserBoundSAS.testOperationWithValidAndExpiredSASToken -- Time elapsed: 2.599 s <<< ERROR!
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemUserBoundSAS.testShouldFailWhenSduoidMismatchesServicePrincipalId -- Time elapsed: 1.504 s <<< ERROR!
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemUserBoundSAS.testBasicOperations -- Time elapsed: 1.515 s <<< ERROR!
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemUserBoundSAS.testCreateFailsWithInvalidOAuthToken -- Time elapsed: 1.651 s <<< ERROR!

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 5
[ERROR] Tests run: 895, Failures: 0, Errors: 5, Skipped: 169
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[ERROR] Tests run: 271, Failures: 1, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[ERROR] org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp.testUpdateDeepDirectoryStructureToRemote -- Time elapsed: 9.415 s <<< FAILURE!

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 732, Failures: 0, Errors: 0, Skipped: 290
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[ERROR] Tests run: 271, Failures: 1, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 892, Failures: 0, Errors: 0, Skipped: 234
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 740, Failures: 0, Errors: 0, Skipped: 147
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 729, Failures: 0, Errors: 0, Skipped: 292
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 737, Failures: 0, Errors: 0, Skipped: 159
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 732, Failures: 0, Errors: 0, Skipped: 205
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 766, Failures: 0, Errors: 0, Skipped: 232
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 729, Failures: 0, Errors: 0, Skipped: 289
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 7s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 42m 45s trunk passed
+1 💚 compile 0m 45s trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 46s trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 checkstyle 0m 34s trunk passed
+1 💚 mvnsite 0m 52s trunk passed
-1 ❌ javadoc 0m 42s /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javadoc 0m 39s /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌ spotbugs 1m 24s /branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings.
+1 💚 shadedclient 31m 30s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 39s the patch passed
+1 💚 compile 0m 35s the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 35s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 23s the patch passed
+1 💚 mvnsite 0m 41s the patch passed
-1 ❌ javadoc 0m 30s /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌ javadoc 0m 29s /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚 spotbugs 1m 23s the patch passed
+1 💚 shadedclient 31m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 3m 19s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
123m 48s
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/1/artifact/out/Dockerfile
GITHUB PR #8141
JIRA Issue HADOOP-19472
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 721d27a0ba48 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 6ba355b
Default Java Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/1/testReport/
Max. process+thread count 599 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/1/console
versions git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is not a part of change here but since we are doing code cleanup here, I feel we should keep these constants in some constant file only like InternalConstants or MetricsConstants.
This file mainly contains constants used in configuration.

Starting from ZERO to SCALE_DIRECTION_NO_ACTION_NEEDED

* Number of bytes in a gigabyte.
*/
public static final long BYTES_PER_GIGABYTE = 1024L * 1024 * 1024;
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few constants in AbfsHttpConstants file that also need to be removed.

@@ -1,107 +0,0 @@
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sine there is only one resource utilization metric left. Do we need the base class?
Can we simply have one stand alone class called ResourceUtilizationMetrics instead of saying read/write?

Also anyway we are monitoring JVM resources here, so single metric is sufficient IMO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants