Skip to content

test, ci: restore torch's flag proxies by assignment (pytest 9.2), and pin pytest - #917

Merged
vasilevklart merged 3 commits into
mainfrom
users/kirill/ci-pytest92
Oct 28, 2026
Merged

test, ci: restore torch's flag proxies by assignment (pytest 9.2), and pin pytest#917
vasilevklart merged 3 commits into
mainfrom
users/kirill/ci-pytest92

Conversation

@vasilevklart

@vasilevklart vasilevklart commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Main's CPU Core went red when CI picked up pytest 9.2.0 (today, 21:30 UTC). It isn't #836: the same tree passes on 9.1.1.

pytest 9.2's monkeypatch restores an attribute that wasn't in the object's __dict__ by deleting it. torch._inductor.config and torch.backends.cuda.matmul serve their flags through __getattr__, so a test's teardown deleted the flag for the rest of the worker: emulate_precision_casts disappears (the FLUX.2 and four Wan2.2 tests after it fail), and the cuBLAS proxy errors on teardown (the watermark test).

A torch_flag fixture in test/modular/conftest.py sets these flags and restores them by assignment; the four call sites use it.

CI now pins pytest==9.2.0 in the CPU Core, dynamo-smoke and rust-transport jobs (and scripts/ci/install.sh), as it already pins torch, triton, pytest-xdist and diffusers. pyproject.toml's floor is unchanged.

Tests

CI's install and command on laude (torch 2.10.0+cpu, pytest 9.2.0):

main this PR
FLUX.2 file, then Wan2.2 file 5 failed 59 passed
test_audio_watermark.py 1 error 42 passed
  • CPU Core: only the 5 tests that also fail on main from laude's environment (e2e validators, eval compare, an uncached Whisper checkpoint). Passes on pytest 9.1.1 too.
  • ruff check clean.

@vasilevklart
test: restore torch's flag proxies by assignment, not monkeypatch (py…
e0744f7 
…test 9.2)

pytest 9.2 records the previous value of a monkeypatched attribute from the
instance __dict__ when no data descriptor is in the way, and undo() deletes
the attribute when there was none. torch._inductor.config and
torch.backends.cuda.matmul serve their flags through __getattr__/__setattr__,
so each undo deleted the flag for the rest of the process: the Inductor
config lost emulate_precision_casts (failing the Wan2.2 fast-decode and
FLUX.2 rounding tests that ran later on the same worker), and the cuBLAS
proxy refused the delete at teardown.

A torch_flag fixture sets such a flag and restores it by assignment. The
torch.backends.cudnn flags are data descriptors and keep using monkeypatch.
@vasilevklart

vasilevklart commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator Author

Feel free to change / add to the branch, not sure if we should pin pytest in the CI

@mstar-review

mstar-review Bot commented Oct 27, 2026

Copy link
Copy Markdown

Automated review

🟢 No findings on e0744f7, light review

What the review checked
  • torch_flag in test/modular/conftest.py saves each value with getattr before setting it, then restores the values in reverse order by assignment at teardown, so the same flag patched twice in one test ends up back at its original value.
  • All four call sites the description names now go through torch_flag: the watermark matmul TF32 flag, the fastconformer matmul TF32 flag, and both FLUX.2 emulate_precision_casts patches.
  • Searched pr-head for other monkeypatch calls on torch._inductor.config, torch._dynamo.config or torch.backends.cuda.matmul, in both object and dotted-string form, and found none left.
  • The remaining monkeypatch.setattr(torch.backends.cudnn, ...) calls were left alone, which is consistent with the PR's results: test_audio_watermark.py also patches cudnn and passes on pytest 9.2 in the reported run. That is expected because cudnn's flags are descriptors defined on the class, not values served through a module __getattr__.
  • torch_flag lives in test/modular/conftest.py and all of its users are under test/modular, so the fixture is in scope wherever it is used.

Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.

dghosef
dghosef approved these changes Oct 27, 2026
View reviewed changes

@dghosef dghosef left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks

nit: the torch.backends.cudnn patches (allow_tf32 / benchmark) are still plain monkeypatch - CI's fine so 9.2 seems to restore those correctly, but might be worth routing through torch_flag too so nobody has to wonder

@vasilevklart
ci: pin pytest to 9.2.0 like the rest of the test stack
68e67aa 
CI pinned torch, triton, pytest-xdist and diffusers but took whatever
pytest was newest, so 9.2's monkeypatch change reached every PR's CPU
Core at once. The three jobs and the Mac mirror in scripts/ci/install.sh
now install pytest 9.2.0; pyproject's floor for developers is unchanged.
@vasilevklart vasilevklart changed the title test: restore torch's flag proxies by assignment, not monkeypatch (pytest 9.2) test, ci: restore torch's flag proxies by assignment (pytest 9.2), and pin pytest Oct 27, 2026
@vasilevklart
test: the cudnn flag patches go through torch_flag too
f556978 
cudnn's flags are class descriptors, which monkeypatch still restores, but
one fixture for every torch flag means nobody has to know which kind a flag
is. cosmos3's windowed-decode test is outside test/modular, where the
fixture lives, and keeps monkeypatch.
@vasilevklart

vasilevklart commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Done

@vasilevklart
vasilevklart merged commit 6d0a86e into main Oct 28, 2026
6 checks passed
@vasilevklart
vasilevklart deleted the users/kirill/ci-pytest92 branch October 28, 2026 00:19
@NSagan271 NSagan271 mentioned this pull request Oct 28, 2026
Merged
dghosef added a commit that referenced this pull request Oct 28, 2026
@dghosef @claude
Merge origin/main 73c6c17 into merge/858-main
636e2b8 
Main brought (since 6cfe39c): #892 Parakeet and Nemotron ASR with the
transducer as a graph loop (and the shared mrope_position_ids helper),
#917 pytest 9.2 flag proxies, #897 one captured all-to-all for Ulysses,
#915 NCCL graph registration off by default, #836 Qwen3-Omni Thinker
prefix caching (ARNodeSubmodule.cuts_tensor_inputs, marks_dynamic_dims,
the conductor's _streamed_partitions), #909 bench cells, #900 Command A+
batched image tiles.

Conflict:
- mstar/model/submodule_base.py: main's ARNodeSubmodule.cuts_tensor_inputs
  beside the stack's token_stop and its stop checks. Independent class
  attributes; both kept.

Ported onto the stack's mechanisms:
- mstar/model/parakeet/submodules.py: check_stop_batched drops the
  step_walk parameter (the stack's three-argument signature) and reads
  its "active" flags through HostRows.first_values instead of its own
  row-map loop. Its stop is a flag, not a token, so it has no TokenStop.
- test/modular/test_transducer_loop.py: the call without step_walk.

The conductor's _streamed_partitions (#836) sits beside _pick_replicas /
_register_request; it reads only partition_states and _resource_specs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dghosef added a commit that referenced this pull request Oct 28, 2026
@dghosef @claude
Merge merge/858-main (origin/main 73c6c17) into merge/859-main
5dada82 
Brings #858's merge of main: #892 Parakeet / Nemotron ASR, #917, #897
Ulysses all-to-all, #915, #836 Qwen3-Omni Thinker prefix caching, #909,
#900.

Conflicts:
- docs/adding_models.rst: main's marks_dynamic_dims sentence and #859's
  compile_regions paragraph are independent facts; both kept, in one
  paragraph.
- mstar/model/qwen3_omni/submodules.py (3 hunks, plus auto-merged hunks
  in the same class): AudioEncoderSubmodule. Two mechanisms for one cause
  (the HF encoder recompiling per clip length): #859 runs it eagerly
  (disable_torch_compile) and packs a step's clips into calls of at most
  MAX_BATCH_FRAMES; #836 builds the encoder's chunk tensors in
  prepare_inputs with mark_dynamic so the per-clip forward compiles once
  (marks_dynamic_dims, _max_seqlen). One mechanism: #859's class verbatim.
  Under disable_torch_compile the engine never compiles the encoder, so
  #836's concern (a recompiled graph rounding differently from the one
  whose output the prefix cache holds) cannot arise. The auto-merge had
  taken #836's kwargs-only forward beside #859's forward_batched, which
  calls forward with the clips positionally; that keep-both is undone.
  Main's VisionEncoderSubmodule (marks_dynamic_dims, prepared indices) and
  Thinker changes (cuts_tensor_inputs, split_inputs, mrope_position_ids,
  image_span) are kept; #859 did not change those. Imports: both
  cp_embed_accumulate and mrope_position_ids; the audio helpers only
  #836's prepare_inputs used are dropped.

Tests following from it (test/modular/test_qwen3_omni_prefix_cache.py):
- the test that #836's prepared chunk tensors embed a clip as the encoder
  does alone is deleted with that code;
- the compile test now asserts the audio encoder stays eager while its
  output is cached;
- the CUDA no-recompile test keeps its vision case only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vasilevklart added a commit that referenced this pull request Oct 28, 2026
@vasilevklart
test: the recompile-limit tests set dynamo's flag through torch_flag …
7c8ec82 
…(pytest 9.2)

pytest 9.2's monkeypatch undo deletes an attribute it didn't find in the
instance __dict__. On torch._dynamo.config that removed recompile_limit
for the rest of the worker, so a GLM-5.2 capture-config test that ran
later on it failed with AttributeError. On a submodule built with
object.__new__, the delete itself raised in nn.Module.__delattr__ at
teardown. The flag now goes through #917's torch_flag fixture, and the
throwaway submodule takes its stub by plain assignment.
dghosef added a commit to dghosef/mstar that referenced this pull request Oct 28, 2026
@dghosef @claude
Merge merge/859-main (mstar-project#859 and mstar-project#858 with or…
55f97af 
…igin/main 73c6c17) into merge/860-main

Brings main's mstar-project#892 Parakeet / Nemotron ASR, mstar-project#917, mstar-project#897 Ulysses as one
captured all-to-all (prefer_all_gather and the all-gather exchange gone,
Cosmos3 captures every tier under TP x SP), mstar-project#915, mstar-project#836 Qwen3-Omni Thinker
prefix caching, mstar-project#909 and mstar-project#900, with mstar-project#858's and mstar-project#859's resolutions.

No conflicts. Files both sides touched (docs/clients.rst,
docs/installation.rst, docs/models.rst, sequence_parallel.py,
cosmos3/submodules.py, test_accelerator_graph_capture.py,
test_flux2_klein_model.py): each merged change equals mstar-project#859's own delta
since 3166401, so nothing kept both sides. mstar-project#860's my_rows / gather_rows
(Wan2.2's CFG split) sit beside main's new ulysses_attention and use
CommGroup.all_gather, which mstar-project#897 left unchanged; nothing in mstar-project#860 passed
prefer_all_gather.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dghosef dghosef dghosef approved these changes

Assignees

No one assigned

Labels

area/infra size/XS type/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vasilevklart @dghosef