Refactor AWS client metrics implementation#14327
Conversation
plugin/trino-hive/src/main/java/io/trino/plugin/hive/aws/AwsApiCallStats.java
Outdated
Show resolved
Hide resolved
plugin/trino-hive/src/main/java/io/trino/plugin/hive/s3select/TrinoS3ClientFactory.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
In case of GlueHiveMetastore the stats field is instantiated per instance, because the metastore can be part of different connectors and the stats can be therefore connected per catalog.
See the outcome of this design choice in the following test:
There was a problem hiding this comment.
Can we make sure through a product test that the s3 stats are also collected per catalog?
There was a problem hiding this comment.
The per-instance instantiation hasn't changed here, so that will be the same as before. Looking around, I don't see any product tests that use a Glue catalog- which is necessary for the JMX object to be registered for such a test. I would add one if it seemed quick, but I'm not sure about how the product test environments get set up or initialized so I would need some help to get the skeleton set up to add a test here.
There was a problem hiding this comment.
Nevermind. By going through the product tests environments, I noticed that the project doesn't have at the moment a product test environment that can be used for AWS tests.
cc @findepi
b4af679 to
0f9b90c
Compare
|
@findepi / @findinpath - are we ok with the current state of the PR to merge it? |
|
@pettyjamesm I find the implementation clean. Thank you for putting up the PR. The only thing I'm not keen of is the seemingly unused method in the |
0f9b90c to
ae2f0fb
Compare
|
@findinpath - ok, rebased and pushed a new commit without the extra method. Let me know if you want anything else changed before you're comfortable approving the PR. |
ae2f0fb to
15c895b
Compare
findinpath
left a comment
There was a problem hiding this comment.
Thank you for the submission and for addressing the comments.
15c895b to
25f5269
Compare
|
@findepi - any additional changes necessary before merging this PR? |
|
@alexjo2144 it would be nice if you could help here. |
|
I don't think the integration tests around JMX are very good. Mind just confirming that you double checked that the stats are still present with the same names? |
I have indeed confirmed the stats are still present with the same names after this refactor. |
25f5269 to
d7abf90
Compare
|
@alexjo2144 - anything else you need in order to merge this PR? |
|
thank you @pettyjamesm @alexjo2144 |
|
Thanks! |
Description
Refactors AWS client stats for S3 and Glue to to consolidate implementations for core SDK metrics reported by all clients via their
RequestMetricCollector. This involves:TimeStat/CounterStat/AtomicLongfields fromTrinoS3FileSystemStatsandGlueMetastoreStatsinto a newAwsSdkClientCoreStatsclass, and embedding an instance of that class into each with the@Flattenannotation to preserve the same fields.AbstractAwsSdkRequestMetricsCollectorand separate subclasses for S3 and Glue in favor of a single shared implementation ofAwsSdkClientCoreRequestMetricCollectorbound to a givenAwsSdkClientCoreStatsGlueMetastoreApiStatsclass to a more generalAwsApiCallStatsclass so that it can be used from other client metrics collectors in the future (none currently implemented in Trino). This is simply a rename and relocation of the class.TimeStatmetrics via the newTimeStat#addNanosmethod which avoids unnecessary time unit conversions and associated floating point precision loss opportunities.Non-technical explanation
No non-technical explanation for this change should be necessary.
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: