Added metrics for restclient and workqueue

[Inode1]

This PR introduces additional metrics to improve observability around the Envoy Gateway, specifically focusing on the REST client and the workqueue.

Motivation
There are currently several issues that limit our ability to effectively monitor and troubleshoot the system:

Lack of Metrics: The limited set of existing metrics makes it difficult to gain insights into the performance and health of the Envoy Gateway. This PR enhances observability and makes it easier to identify and diagnose performance issues—particularly those related to the workqueue.

Workqueue Metrics Conflict: Metrics for the workqueue are registered both by controller-runtime and k8s.io/component-base via init() functions. However, only one of these libraries can successfully register the metrics. In our setup, k8s.io/component-base wins, which results in the metrics from controller-runtime not being exposed at all.

This PR resolves these issues by explicitly adding and managing the necessary metrics, ensuring better visibility into queue behavior and client interactions.

Release Notes: Yes

[zirain]

can you add a test/e2e to make sure it worked as expected.

[codecov]

Codecov Report

Attention: Patch coverage is 15.00000% with 51 lines in your changes missing coverage. Please review.

Project coverage is 65.27%. Comparing base (40fc25d) to head (af16e79).
Report is 41 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/metrics/restclient/metrics.go	0.00%	22 Missing ⚠️
internal/metrics/workqueue/metrics.go	0.00%	22 Missing ⚠️
internal/metrics/register.go	0.00%	7 Missing ⚠️

❌ Your patch status has failed because the patch coverage (15.00%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5731      +/-   ##
==========================================
+ Coverage   65.21%   65.27%   +0.05%     
==========================================
  Files         214      217       +3     
  Lines       34321    34666     +345     
==========================================
+ Hits        22384    22628     +244     
- Misses      10584    10660      +76     
- Partials     1353     1378      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Inode1]

/retest

[Inode1]

@zirain, could you take a look?

[arkodg]

can you share how this is different from https://github.com/kubernetes-sigs/controller-runtime/blob/6ad5c1dd4418489606d19dfb87bf38905b440561/pkg/internal/metrics/workqueue.go#L23

which is being set by default
https://github.com/kubernetes-sigs/controller-runtime/blob/6ad5c1dd4418489606d19dfb87bf38905b440561/pkg/controller/priorityqueue/priorityqueue.go#L58

[arkodg]

reread the title description, trying to better understand the conflict

[Inode1]

Sure, this and this both attempt to initialize workqueue metrics. But only one provider can be registered — enforced here in client-go. Component-base wins in our case. EG dont use PQ

[arkodg]

thanks