HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus #4665

qijiale76 · 2022-08-01T11:15:28Z

Description of PR

HDFS-16302 intoduced request record for each namespace, but it is only exposed in /jmx endpoint and in json format, not very convenient.
this patch exposed these metrics in /prom endpoint for Prometheus

How was this patch tested?

manual testing

hadoop-yetus · 2022-08-01T13:49:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 58s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 48s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 7s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 41s		trunk passed
+1 💚	shadedclient	23m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 42s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 36s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 22s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 37s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	1m 27s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	shadedclient	23m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	30m 46s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		152m 41s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	The method name org.apache.hadoop.hdfs.server.federation.metrics.FederationRPCPerformanceMonitor.ProxyOpPermitAccepted(String) doesn't start with a lower case letter At FederationRPCPerformanceMonitor.java:start with a lower case letter At FederationRPCPerformanceMonitor.java:[lines 201-205]
	The method name org.apache.hadoop.hdfs.server.federation.metrics.FederationRPCPerformanceMonitor.ProxyOpPermitRejected(String) doesn't start with a lower case letter At FederationRPCPerformanceMonitor.java:start with a lower case letter At FederationRPCPerformanceMonitor.java:[lines 189-197]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/1/artifact/out/Dockerfile
GITHUB PR	#4665
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 96b1594e0e4a 4.15.0-184-generic #194-Ubuntu SMP Thu Jun 2 18:54:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e4e0142`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/1/testReport/
Max. process+thread count	2271 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

goiri · 2022-08-01T17:43:28Z

...hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcMonitor.java

+  /**
+   * Rejected to proxy an operation to a Namenode.
+   */
+  void ProxyOpPermitRejected(String nsId);


lower case p?

goiri · 2022-08-01T17:43:39Z

...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java

                " is overloaded for NS: " + nsId;
        throw new StandbyException(msg);
      }
+      if (rpcMonitor!= null) {


...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java

hadoop-yetus · 2022-08-10T09:02:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 13s		trunk passed
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 48s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 54s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 10s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 42s		trunk passed
+1 💚	shadedclient	24m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 41s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 41s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 36s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 22s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 37s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 26s		the patch passed
+1 💚	shadedclient	23m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 50s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		137m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/2/artifact/out/Dockerfile
GITHUB PR	#4665
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 126ff9bc843e 4.15.0-184-generic #194-Ubuntu SMP Thu Jun 2 18:54:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `514fb4f`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/2/testReport/
Max. process+thread count	2294 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-08-10T12:13:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 3s		trunk passed
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 48s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 42s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	1m 0s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 5s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 42s		trunk passed
+1 💚	shadedclient	23m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 42s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 38s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 27s		the patch passed
+1 💚	shadedclient	23m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 39s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		136m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/3/artifact/out/Dockerfile
GITHUB PR	#4665
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d1dc88b8a8f6 4.15.0-184-generic #194-Ubuntu SMP Thu Jun 2 18:54:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `388fb58`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/3/testReport/
Max. process+thread count	2317 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2023-04-22T09:03:32Z

Looks good, but have triggered the build again to see if things are still calm. If the build comes green will push this in.

hadoop-yetus · 2023-04-22T11:05:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 27s		trunk passed
+1 💚	compile	0m 38s		trunk passed
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	1m 2s		trunk passed
+1 💚	spotbugs	1m 34s		trunk passed
+1 💚	shadedclient	23m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 31s		the patch passed
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 17s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	javadoc	0m 49s		the patch passed
+1 💚	spotbugs	1m 23s		the patch passed
+1 💚	shadedclient	23m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	21m 39s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		122m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/4/artifact/out/Dockerfile
GITHUB PR	#4665
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d39c77316add 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `388fb58`
Default Java	Red Hat, Inc.-1.8.0_362-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/4/testReport/
Max. process+thread count	2121 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4665/4/console
versions	git=2.9.5 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

Changes LGTM

…uest record metrics for each nameservice to Prometheus (apache#4665). Contributed by Jiale Qi. Reviewed-by: Inigo Goiri <[email protected]> Signed-off-by: Ayush Saxena <[email protected]>

HDFS-16707

e4e0142

goiri reviewed Aug 1, 2022

View reviewed changes

fix code style issues & add 'ns' tag

514fb4f

fix code style issues

388fb58

goiri approved these changes Aug 10, 2022

View reviewed changes

ayushtkn approved these changes Apr 26, 2023

View reviewed changes

ayushtkn merged commit 1079890 into apache:trunk May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus #4665

HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus #4665

Uh oh!

qijiale76 commented Aug 1, 2022

Uh oh!

hadoop-yetus commented Aug 1, 2022

Uh oh!

goiri Aug 1, 2022

Uh oh!

goiri Aug 1, 2022

Uh oh!

Uh oh!

hadoop-yetus commented Aug 10, 2022

Uh oh!

hadoop-yetus commented Aug 10, 2022

Uh oh!

ayushtkn commented Apr 22, 2023

Uh oh!

hadoop-yetus commented Apr 22, 2023

Uh oh!

ayushtkn left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus #4665

HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus #4665

Uh oh!

Conversation

qijiale76 commented Aug 1, 2022

Description of PR

How was this patch tested?

Uh oh!

hadoop-yetus commented Aug 1, 2022

Uh oh!

goiri Aug 1, 2022

Choose a reason for hiding this comment

Uh oh!

goiri Aug 1, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hadoop-yetus commented Aug 10, 2022

Uh oh!

hadoop-yetus commented Aug 10, 2022

Uh oh!

ayushtkn commented Apr 22, 2023

Uh oh!

hadoop-yetus commented Apr 22, 2023

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants