Update metrics linting to be able to handle custom metrics #18733

MadLittleMods · 2025-07-24T20:31:06Z

Update metrics linting to be able to handle custom metrics

Dev notes

GaugeBucketCollector
LaterGauge

mypy extensions

Mypy ~~Bug~~ /limitation where multiple extensions using get_base_class_hook interfere with each other python/mypy#19524

Todo

Figure out how to run analyze_prometheus_metric_classes regardless of other mypy plugins (Multiple extensions using get_base_class_hook interfere with each other python/mypy#19524), see Update metrics linting to be able to handle custom metrics #18733 (comment)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

From #18656

``` synapse/metrics/__init__.py:522: error: Expected the `labelnames` argument of Histogram to be a list of label names (including `SERVER_NAME_LABEL`), but got TupleExpr:528( StrExpr(type) StrExpr(reason) NameExpr(SERVER_NAME_LABEL [synapse.metrics.SERVER_NAME_LABEL])). If this is a process-level metric (vs homeserver-level), use a type ignore comment to disable this check. [missing-server-name-label] ```

Conflicts: scripts-dev/mypy_synapse_plugin.py

scripts-dev/mypy_synapse_plugin.py

Conflicts: scripts-dev/mypy_synapse_plugin.py

scripts-dev/mypy_synapse_plugin.py

Conflicts: synapse/metrics/__init__.py

Conflicts: scripts-dev/mypy_synapse_plugin.py

devonh · 2025-07-30T19:53:23Z

I believe this PR looks good to merge as is.
But I think we should discuss with the team first to make sure everyone is in agreement that this makes sense to add.
As long as others are aware that we are intentionally creating a lint that will disappear into oblivion and rarely, if ever, get run, then I am happy to bring it into the codebase.

… plugin See #18733 (comment) Discussed at https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$yA64EMlsudEUdu2tRxQDcsF1HLqblXCOLCX9282asFw?via=matrix.org&via=element.io&via=node.marinchik.ink

scripts-dev/mypy_synapse_plugin.py

See #18733 (comment)

devonh

Thanks for pushing through to get this running by default!

MadLittleMods · 2025-08-01T20:36:16Z

Thanks for the review @devonh and @frebib for the suggestion to make the lint viable 🦛

MadLittleMods added 4 commits July 24, 2025 14:10

Add in base linting for metrics

4df927b

From #18656

Make label name arg position/name generic

d62d564

Update wording

325caec

MadLittleMods added the A-Metrics label Jul 24, 2025

MadLittleMods added 3 commits July 24, 2025 15:34

Explain why prometheus_client.metrics_core.Metric isn't included here

53bfbc6

Disambiguate

395b33b

Remove debug log

7818657

MadLittleMods mentioned this pull request Jul 24, 2025

Refactor metrics to be scoped to the homeserver #18592

Closed

38 tasks

Fix lints in in lint

6bb5fef

MadLittleMods mentioned this pull request Jul 25, 2025

Refactor GaugeBucketCollector metrics to be homeserver-scoped #18715

Merged

3 tasks

Merge branch 'develop' into madlittlemods/18592-lint-metric

b073bb1

Conflicts: scripts-dev/mypy_synapse_plugin.py

MadLittleMods commented Jul 28, 2025

View reviewed changes

scripts-dev/mypy_synapse_plugin.py Outdated Show resolved Hide resolved

MadLittleMods added 8 commits July 28, 2025 17:07

Add cross-check for prometheus_metric_fullname_to_label_arg_map

6758742

Ignore some process-level metrics

c53d34d

Ignore process-level metrics

07e79fe

Fix JemallocCollector edge case

4902046

Support almost all metric types

da38631

Add some cross-links

b46040a

Lint Metric types

8d706ac

Ignore Metric usage where necessary

478edc2

MadLittleMods changed the title ~~Updating metrics linting to be able to handle custom metrics~~ Update metrics linting to be able to handle custom metrics Jul 28, 2025

Add changelog

62ce1e0

MadLittleMods mentioned this pull request Jul 29, 2025

Refactor LaterGauge metrics to be homeserver-scoped #18714

Merged

9 tasks

MadLittleMods added 2 commits July 28, 2025 19:17

Remove irrelevant comment

9166d55

Beef up comment

2d947a4

MadLittleMods commented Jul 29, 2025

View reviewed changes

scripts-dev/mypy_synapse_plugin.py Outdated Show resolved Hide resolved

Mark off TODO about Metric

89076a4

MadLittleMods marked this pull request as ready for review July 29, 2025 00:47

MadLittleMods requested a review from a team as a code owner July 29, 2025 00:47

Merge branch 'develop' into madlittlemods/18592-lint-metric

a90e19b

Conflicts: scripts-dev/mypy_synapse_plugin.py

MadLittleMods commented Jul 29, 2025

View reviewed changes

scripts-dev/mypy_synapse_plugin.py Outdated Show resolved Hide resolved

MadLittleMods added 5 commits July 29, 2025 11:48

Merge branch 'develop' into madlittlemods/18592-lint-metric

44e30cf

Conflicts: synapse/metrics/__init__.py

Merge branch 'develop' into madlittlemods/18592-lint-metric

86a0bcd

Conflicts: synapse/metrics/__init__.py

Uncomment merged lints

ab5a756

Merge branch 'develop' into madlittlemods/18592-lint-metric

8c380c0

Conflicts: scripts-dev/mypy_synapse_plugin.py

We can now use the full list

bec7bfb

MadLittleMods added 3 commits July 31, 2025 17:19

Workaround mypy limitation only calling get_base_class_hook for one…

21ef4b4

… plugin See #18733 (comment) Discussed at https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$yA64EMlsudEUdu2tRxQDcsF1HLqblXCOLCX9282asFw?via=matrix.org&via=element.io&via=node.marinchik.ink

More sound logic

61a6f39

Use lint instead of assert

4965340

MadLittleMods requested review from devonh and removed request for a team July 31, 2025 23:20

devonh reviewed Aug 1, 2025

View reviewed changes

scripts-dev/mypy_synapse_plugin.py Show resolved Hide resolved

MadLittleMods added 3 commits August 1, 2025 12:13

Make sure pydantic mypy hooks are still being called

a7573a7

See #18733 (comment)

Add more explicit instructions

d2efe1d

Merge branch 'develop' into madlittlemods/18592-lint-metric

f2f1b9e

MadLittleMods requested a review from devonh August 1, 2025 17:48

devonh approved these changes Aug 1, 2025

View reviewed changes

MadLittleMods merged commit e16fbdc into develop Aug 1, 2025
44 checks passed

MadLittleMods deleted the madlittlemods/18592-lint-metric branch August 1, 2025 20:34

MadLittleMods mentioned this pull request Aug 1, 2025

Multiple extensions using get_base_class_hook interfere with each other python/mypy#19524

Open

MadLittleMods mentioned this pull request Oct 10, 2025

Update TODO list of conflicting areas where we encounter metrics being clobbered (ApplicationService) #19040

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update metrics linting to be able to handle custom metrics #18733

Update metrics linting to be able to handle custom metrics #18733

Uh oh!

MadLittleMods commented Jul 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devonh commented Jul 30, 2025

Uh oh!

Uh oh!

devonh left a comment

Uh oh!

Uh oh!

MadLittleMods commented Aug 1, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Update metrics linting to be able to handle custom metrics #18733

Update metrics linting to be able to handle custom metrics #18733

Uh oh!

Conversation

MadLittleMods commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev notes

mypy extensions

Todo

Pull Request Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devonh commented Jul 30, 2025

Uh oh!

Uh oh!

devonh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MadLittleMods commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MadLittleMods commented Jul 24, 2025 •

edited

Loading

MadLittleMods commented Aug 1, 2025 •

edited

Loading