[ty] Add separate CI job for memory usage stats #19134

ibraheemdev · 2025-07-03T23:44:39Z

Summary

As discussed in #19059.

ibraheemdev · 2025-07-03T23:45:21Z

crates/ty/src/lib.rs

-                                            ExitStatus::Failure
-                                        } else {
-                                            ExitStatus::Success
+                        if std::env::var("TY_MEMORY_REPORT").as_deref() == Ok("mypy_primer") {


This is a hack for now, we should have a way to silence diagnostics completely (but keep the memory usage report).

ibraheemdev · 2025-07-03T23:45:37Z

crates/ty_python_semantic/resources/primer/memory.txt

+flake8
+sphinx
+prefect
+trio


Somewhat arbitrarily chosen, open to suggestions.

sympy might be a good addition here, I think -- we have very high memory usage on it already

ibraheemdev · 2025-07-03T23:47:57Z

crates/ty_project/src/db.rs

            // the exact numbers across runs and compute the difference, but we don't have
            // the infrastructure for that currently.
-            const BASE: f64 = 1.1;
+            const BASE: f64 = 1.05;


I lowered the threshold because single-threaded runs should be completely deterministic, and the jump between buckets is currently quite high for large numbers.

github-actions · 2025-07-03T23:59:16Z

`mypy_primer` results

No ecosystem changes detected ✅

ibraheemdev · 2025-07-04T00:15:52Z

We could also just run the memory usage report in the same job? If we choose smaller projects that shouldn't add significant time, and we avoid having to use another depot runner and duplicate the setup work. We could also avoid using mypy-primer for that run and get exact numbers.

AlexWaygood

thank you!

.github/workflows/mypy_primer.yaml

github-actions · 2025-07-04T21:25:40Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

AlexWaygood

Nice!

AlexWaygood · 2025-07-05T16:44:35Z

crates/ty_python_semantic/resources/primer/memory.txt

+flake8
+sphinx
+prefect
+trio


sympy might be a good addition here, I think -- we have very high memory usage on it already

MichaReiser · 2025-07-07T08:42:22Z

I'm a bit surprised that the memory usage results are non deterministic when running multi threaded (to such large degrees). The only two reasons I can think of are:

AST GC
intermediate results from fixpoint cycles and the non-determinism comes from entering the cycles from different entry queries.

Do we know which of the two is causing the flakes?

MichaReiser · 2025-07-07T08:45:48Z

crates/ty/src/lib.rs

                            tracing::warn!("No python files found under the given path(s)");
                        }

+                        // TODO: We should have an official flag to silence workspace diagnostics.


I, at first, didn't understand what features is needed to replace this environment variable usage but I think I understand now. What we need is a --quiet or similar flag. Would you mind opening an issue in the ty repository for adding a flag to suppress diagnostic printing (and mention Ruff's --quiet and --silent flags)

Created astral-sh/ty#772

ibraheemdev · 2025-07-07T16:22:00Z

I'm a bit surprised that the memory usage results are non deterministic when running multi threaded (to such large degrees)

This is partly due to our rounding strategy being affected by very small changes if we get unlucky with the numbers. Now that we have a separate job we could just get the exact numbers manually. That said, on my machine I was seeing variance of ~10MB on projects with total memory usage in the hundreds of MB. It's possible that AST garbage collection is the culprit.

## Summary As discussed in #19059.

…c_tokens * 'main' of https://github.com/astral-sh/ruff: (27 commits) [ty] First cut at semantic token provider (astral-sh#19108) [`flake8-simplify`] Make example error out-of-the-box (`SIM116`) (astral-sh#19111) [`flake8-use-pathlib`] Make example error out-of-the-box (`PTH210`) (astral-sh#19189) [`flake8-use-pathlib`] Add autofixes for `PTH203`, `PTH204`, `PTH205` (astral-sh#18922) [`flake8-type-checking`] Fix syntax error introduced by fix (`TC008`) (astral-sh#19150) [`flake8-pyi`] Make example error out-of-the-box (`PYI007`, `PYI008`) (astral-sh#19103) Update Rust crate indicatif to 0.18.0 (astral-sh#19165) [ty] Add separate CI job for memory usage stats (astral-sh#19134) [ty] Add documentation for server traits (astral-sh#19137) Rename to `SessionSnapshot`, move unwind assertion closer (astral-sh#19177) [`flake8-type-checking`] Make example error out-of-the-box (`TC001`) (astral-sh#19151) [ty] Bare `ClassVar` annotations (astral-sh#15768) [ty] Re-enable multithreaded pydantic benchmark (astral-sh#19176) [ty] Implement equivalence for protocols with method members (astral-sh#18659) [ty] Use RHS inferred type for bare `Final` symbols (astral-sh#19142) [ty] Support declaration-only attributes (astral-sh#19048) [ty] Sync vendored typeshed stubs (astral-sh#19174) Update dependency pyodide to ^0.28.0 (astral-sh#19164) Update NPM Development dependencies (astral-sh#19170) Update taiki-e/install-action action to v2.56.7 (astral-sh#19169) ...

ibraheemdev added ci Related to internal CI tooling ty Multi-file analysis & type inference labels Jul 3, 2025

ibraheemdev commented Jul 3, 2025

View reviewed changes

ibraheemdev force-pushed the ibraheem/mypy-primer-determinism branch from a6cc106 to 81a95cc Compare July 3, 2025 23:55

ibraheemdev force-pushed the ibraheem/mypy-primer-determinism branch 2 times, most recently from a153f65 to eba3b35 Compare July 4, 2025 00:14

ibraheemdev force-pushed the ibraheem/mypy-primer-determinism branch from eba3b35 to bf4df9e Compare July 4, 2025 00:27

ibraheemdev marked this pull request as ready for review July 4, 2025 00:32

ibraheemdev requested review from AlexWaygood, MichaReiser, carljm, dcreager and sharkdp as code owners July 4, 2025 00:32

AlexWaygood reviewed Jul 4, 2025

View reviewed changes

.github/workflows/mypy_primer.yaml Outdated Show resolved Hide resolved

ibraheemdev force-pushed the ibraheem/mypy-primer-determinism branch 7 times, most recently from 950612b to a941e5e Compare July 4, 2025 21:14

add separate CI job for memory usage stats

6f85184

ibraheemdev force-pushed the ibraheem/mypy-primer-determinism branch from a941e5e to 6f85184 Compare July 4, 2025 21:26

AlexWaygood approved these changes Jul 5, 2025

View reviewed changes

MichaReiser reviewed Jul 7, 2025

View reviewed changes

MichaReiser approved these changes Jul 7, 2025

View reviewed changes

ibraheemdev merged commit cd84898 into main Jul 7, 2025
37 checks passed

ibraheemdev deleted the ibraheem/mypy-primer-determinism branch July 7, 2025 16:17

AlexWaygood pushed a commit that referenced this pull request Jul 7, 2025

[ty] Add separate CI job for memory usage stats (#19134)

4cdd84d

## Summary As discussed in #19059.

AlexWaygood mentioned this pull request Jul 7, 2025

[ty] Do not consider a type T to satisfy a method member on a protocol unless the method is available on the meta-type of T #19187

Merged

[ty] Add separate CI job for memory usage stats #19134

[ty] Add separate CI job for memory usage stats #19134

Uh oh!

Conversation

ibraheemdev commented Jul 3, 2025

Summary

Uh oh!

ibraheemdev Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

ibraheemdev Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Jul 5, 2025

Choose a reason for hiding this comment

Uh oh!

ibraheemdev Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

ibraheemdev commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Jul 5, 2025

Choose a reason for hiding this comment

Uh oh!

MichaReiser commented Jul 7, 2025

Uh oh!

MichaReiser Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

ibraheemdev Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ibraheemdev commented Jul 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Jul 3, 2025 •

edited

Loading

`mypy_primer` results

ibraheemdev commented Jul 4, 2025 •

edited

Loading

github-actions bot commented Jul 4, 2025 •

edited

Loading

`ruff-ecosystem` results