[merceod]

V-JEPA2 embeddings served by M* sat at cosine 0.83-0.89 against the HF fp32 model, while HF's own bf16 run sits at 0.94-0.98. The gap was preprocessing, not the model.

_preprocess_video resized the frames as floats with interpolate, and HF's VJEPA2VideoProcessor resizes the uint8 frames with antialias and rounds. On an upscale like DROID's 320x180 the two differ by at most one uint8 step per pixel, and V-JEPA2 is that sensitive. The HF fp32 model fed our pixels scores 0.838-0.944 against the same model on its own processor's pixels, which is the served gap to the digit. On a downscale the float resize also skips antialiasing, up to 54 uint8 steps off at 480x640.

This PR runs the checkpoint's own processor. load_video returns the uint8 frames as decoded, and process_prompt hands them to the VJEPA2VideoProcessor that AutoVideoProcessor loads from video_preprocessor_config.json (crop size, mean and std, crop checked against config.json), like Qwen3.5 does. The AC model keeps the official repo's central square crop from #921, now also from the bytes, one frame's square in float at a time, bit-identical to upstream's transform. No float copy of the full-size clip is made.

This changes the outputs of vjepa2 and vjepa2_ac on every platform, toward the reference.

64 frames of 1080p on CPU, from the decoder's uint8 frames to video_frames. Peak RSS above the idle process, median of 15 runs, 4 threads is the API server's default preprocess pool.

	peak RSS	4 threads	1 thread
main	2.0 GB	0.39 s	1.36 s
uint8 round trip (c0fffe5)	4.8 GB	1.53 s	4.25 s
this PR	0.9 GB	0.31 s	0.82 s

test/modular/vjepa2/test_preprocess.py runs the served path and HF's processor on synthetic clips at four sizes and asserts they match within 1e-4. The old code fails all four, and "within one uint8 step" would not have caught this. It also covers load_video handing over the decoded bytes, a processor whose crop disagrees with the config, and the AC crop against upstream's float transform.

Command	Result
ruff check .	clean
pytest test/modular/vjepa2/test_preprocess.py	17 passed, on transformers 4.57.3 and 5.22.1
CPU Core paths, CI's selection	10331 passed, 48 failed, the same 48 as main in this env
vjepa2 served on one MI300X vs HF fp32, 10 DROID clips, earlier version with the same pixels	min 0.942 / median 0.978 (was 0.833 / 0.893)
vjepa2 served on one H200 vs HF fp32, same clips	min 0.975 / median 0.977 (main 0.888 / 0.903, HF bf16 itself 0.972 / 0.974), two passes identical

[mstar-review]

[P2] test/cpu-core.txt has been deleted on main, so this hunk is a modify/delete conflict. Resolving it by keeping the file breaks CI.

On current main, CPU Core collects every module under CPU_TEST_PATHS (scripts/ci/select_cpu_tests.py), so test/modular/vjepa2/test_preprocess.py runs without any manifest entry. If the rebase keeps the file, test/ci/test_cpu_selection.py::test_no_hand_kept_cpu_test_list fails because the file exists again. Drop this hunk when rebasing.

[merceod]

/review

[mstar-review]

Automated review

🟡 No findings on c0fffe5, light review

Description

• The description says the new test is registered "in test/cpu-core.txt" and lists python -m pytest @test/cpu-core.txt, but that file doesn't exist on main and the diff doesn't add it. CPU Core now selects tests by directory through CPU_TEST_PATHS in scripts/ci/select_cpu_tests.py. The test still runs because it is under test/modular/, but the registration claim and the command should be updated, and the base comparison rerun on current main.
• The served-embedding numbers (vjepa2 on MI300X, and the vjepa2_ac step-0 latent cosines) don't give the torch/ROCm build or the command or harness that produced them, and the AC rollout result names no command at all.
• It doesn't say whether the GPU integration tests test/integration/test_vjepa2.py and test/integration/test_vjepa2_ac.py were rerun. Both outputs change by design, so it should say they were rerun and passed, or why they weren't.

What the review checked

• _preprocess_video and _preprocess_video_droid are only called from process_prompt, so no other caller depends on the old float-resize output or its dtype.
• load_video returns frames.float() / 255.0, so (frames * 255).round().clamp(0, 255).to(uint8) reproduces the decoded uint8 pixels exactly.
• torchvision is a core dependency in pyproject.toml (torchvision>=0.24.1), so the lazy torchvision.transforms.v2 import in the serving path can't fail on a standard install.
• predictor_kind is set on the config in _load_config, and VJepa2ACModel hard-codes it to "ac", so the DROID branch is chosen only for the AC model.
• The AC transform matches benchmark/vjepa2_repo_ac.py, which calls upstream make_transforms with scale and aspect ratio 1, consistent with the description.
• In the DROID edge-row test, with bilinear interpolation and align_corners=False, output row 0 clamps to input row 0 and rows 2 and up sample only the grey interior, so its assertions hold for both 180x320 and 320x180.
• The new test module sits in test/modular/vjepa2/, which has an __init__.py, so its test_preprocess.py basename doesn't collide with test/modular/test_preprocess_warmup.py. Its optional imports are guarded with importorskip, so it passes or skips on a CPU-only machine.
• Model code allocates no resources, captures no graphs and syncs no streams, so invariants 1-3 and 7 aren't affected: the change is confined to host-side preprocessing in process_prompt.

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

[Gaurav-Shah05]

The pixels now match HF's processor, but the uint8 round trip makes preprocessing a large clip slower than on main (see the inline comment).

[Gaurav-Shah05]

This undones the / 255 at the end of VJepa2Model.load_video. At 64 frames of 1080p it allocates three float32 copies of the clip, 1.6 GB each, before the resize, which is far more memory traffic than main's whole resize.

Doing what Qwen3.5 does would remove it:

• load_video returns the uint8 frames
• _preprocess_video calls the checkpoint's processor through AutoVideoProcessor, which reads the crop size, mean and std from video_preprocessor_config.json

[merceod]

Good catch, thanks. Changed it the way you suggested.

• load_video returns the uint8 frames as decoded, no / 255
• the HF checkpoints go through their own processor via AutoVideoProcessor, so crop size, mean and std come from video_preprocessor_config.json. Its crop has to match config.json, and it loads in warmup_preprocess
• the AC model keeps the official repo's central square crop (on main since V-JEPA2-AC: take the central square crop the official repo uses #921), now also from the bytes, one frame's square in float at a time. Same floats as upstream's transform, bit for bit

64 frames of 1080p on CPU, from the decoder's uint8 frames to video_frames (load_video + process_prompt). Peak RSS above the idle process, median of 15 runs, 4 threads is the API server's default preprocess pool.

	peak RSS	4 threads	1 thread
main	2.0 GB	0.39 s	1.36 s
c0fffe5	4.8 GB	1.53 s	4.25 s
now	0.9 GB	0.31 s	0.82 s

The peaks include the 0.4 GB of decoded frames. The AC path at the same size goes from 2.0 GB and 0.37 s on main to 0.46 GB and 0.16 s. Pixels still match HF's processor at the four test sizes. Main is up to 54 uint8 steps off at 480x640, where its float resize skips antialiasing.

I merged main into the branch instead of rebasing, so your review is still in the history. Served numbers on an H200 coming next.

[merceod]

Served numbers on one H200 for 82a9b62. vjepa2 (facebook/vjepa2-vitl-fpc64-256), the same 10 DROID clips, against HF's VJEPA2Model on the same decoded frames through HF's processor. Cosine over each clip's whole output.

	encoder, min / median	encoder + predictor, min / median
this PR vs HF fp32	0.975 / 0.977	0.9992 / 0.9995
main vs HF fp32	0.888 / 0.903	0.985 / 0.990
HF bf16 vs HF fp32	0.972 / 0.974	0.9992 / 0.9994

Two passes over the clips give identical outputs.

Preprocessing per request served (mstar_request_preprocess_seconds, decode included) goes from 50 to 24 ms on the DROID clips and from 1.87 to 0.77 s on a 64-frame 1080p clip. The API server's peak RSS during the 1080p requests goes from 5.7 to 2.3 GB.

test/integration/test_vjepa2.py passes on the same GPU. test_vjepa2_ac.py skips there since the upstream repo isn't installed.

[Gaurav-Shah05]

LGTM, with one small point below.

[Gaurav-Shah05]

A checkpoint without video_preprocessor_config.json hits a bug that main doesn't have: warm_up_model only logs what warmup_preprocess raises, so the server starts and then fails every video request. Loading the processor in __init__, beside _load_config, refuses such a checkpoint at startup, along with one whose processor crops another size.

[merceod]

Done in 3be6ad4, thanks. The processor now loads in __init__ beside _load_config, so a checkpoint without video_preprocessor_config.json, or with another crop, is refused at startup (tests in 211215a).