Skip to content

Expose AWS Glue metastore stats via JMX#13087

Merged
ebyhr merged 2 commits intotrinodb:masterfrom
findinpath:expose-glue-hive-metastore-stats
Jul 12, 2022
Merged

Expose AWS Glue metastore stats via JMX#13087
ebyhr merged 2 commits intotrinodb:masterfrom
findinpath:expose-glue-hive-metastore-stats

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

@findinpath findinpath commented Jul 5, 2022

Description

Expose the Glue Hive Metastore stats over JMX.
Even though the Glue stats were previously collected, due to a regression issue, they weren't exposed by JMX.

The stats are exposed in JMX under the following object names:

  • hive: "trino.plugin.hive.metastore.glue:type=GlueHiveMetastore,name=hive"
  • iceberg: "io.trino.plugin.hive.metastore.glue:type=GlueMetastoreStats,name=iceberg"
  • delta: "io.trino.plugin.hive.metastore.glue:type=GlueHiveMetastore,name=delta"

Is this change a fix, improvement, new feature, refactoring, or other?

Fix

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Hive connector

How would you describe this change to a non-technical end user or system administrator?

Expose the Glue Hive Metastore stats over JMX.
In Trino, the users could issue a statement like the following to retrieve the AWS Glue stats in the connectors backed by AWS Glue.

For Hive:

select * from jmx.current."trino.plugin.hive.metastore.glue:name=hive,type=gluehivemetastore";

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Hive, Iceberg, Delta Lake
* Expose AWS Glue metastore stats via JMX

@losipiuk
Copy link
Copy Markdown
Member

losipiuk commented Jul 5, 2022

nit: Expose AWS Glue metastore stats on JMX -> Expose AWS Glue metastore stats via JMX

@findinpath findinpath force-pushed the expose-glue-hive-metastore-stats branch from e8af62c to 7c818a0 Compare July 5, 2022 18:40
@findinpath findinpath changed the title Expose Glue Hive Metastore JMX statistics Expose AWS Glue metastore stats via JMX Jul 5, 2022
@raunaqmorarka
Copy link
Copy Markdown
Member

Can we add a test ?

@findinpath findinpath self-assigned this Jul 6, 2022
@findinpath
Copy link
Copy Markdown
Contributor Author

findinpath commented Jul 7, 2022

Can we add a test ?

@raunaqmorarka Due to the fact that Trino does not currently have a a product test environment specific for Glue (see #5426) for Hive / Iceberg, I can create a product test only for Delta Lake.

Integration Tests depend on the TestingTrinoServer configured with TestingJmxModule which does not catch the beans registered in the connectors (which depend on MBeanServerModule and subsequently on ManagementFactory.getPlatformMBeanServer()). This is why, the MBeans registered in the integration tests are not visible through getQueryRunner()).getCoordinator().getMbeanServer().

@findinpath findinpath force-pushed the expose-glue-hive-metastore-stats branch 2 times, most recently from 8ec95f2 to f0960d6 Compare July 7, 2022 13:21
@findinpath findinpath requested review from ebyhr and losipiuk July 7, 2022 13:21
@findinpath findinpath force-pushed the expose-glue-hive-metastore-stats branch from f0960d6 to 840f2f2 Compare July 7, 2022 14:03
The stats are exposed in JMX under the following object names:

- hive: "trino.plugin.hive.metastore.glue:type=GlueHiveMetastore,name=hive"
- iceberg: "io.trino.plugin.hive.metastore.glue:type=GlueMetastoreStats,name=iceberg"
- delta: "io.trino.plugin.hive.metastore.glue:type=GlueHiveMetastore,name=delta"
@ebyhr ebyhr force-pushed the expose-glue-hive-metastore-stats branch from 840f2f2 to da21bea Compare July 12, 2022 09:05
@ebyhr ebyhr merged commit 1fde1ac into trinodb:master Jul 12, 2022
@ebyhr
Copy link
Copy Markdown
Member

ebyhr commented Jul 12, 2022

Merged, thanks!

@github-actions github-actions bot added this to the 390 milestone Jul 12, 2022
@ebyhr ebyhr mentioned this pull request Jul 12, 2022
}

@Managed
@Flatten
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dain you removed these two annotations in a commit here: a4142de#diff-d827fd361f2fef94d35ef8593d0f9a7f168366f28acdb88a67ff413b1904a3d0

Was that on purpose?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

5 participants