Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reenable OpInfo tests. #8519

Merged
merged 1 commit into from
Dec 23, 2024
Merged

Reenable OpInfo tests. #8519

merged 1 commit into from
Dec 23, 2024

Conversation

qihqi
Copy link
Collaborator

@qihqi qihqi commented Dec 23, 2024

When we updated torch version to 2.6, one behavior of test_ops changed.

This line instantiate_device_type_tests(TestOpInfo, globals(), only_for='cpu') now treats only_for argument as iteratable instead of one string. so 'cpu' is split into ['c', 'p', 'u']; and this test did not run.

Running pytest test/test_ops.py gives

(xla2) hanq@hanq-compile-2:/mnt/hanq/pytorch/xla/experimental/torch_xla2$ pytest test/test_ops.py
/mnt/hanq/miniconda3/envs/xla2/lib/python3.10/site-packages/pytest_benchmark/logger.py:46: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
  warner(PytestBenchmarkWarning(text))
=================================== test session starts ====================================
platform linux -- Python 3.10.14, pytest-8.3.2, pluggy-1.5.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /mnt/hanq/pytorch/xla/experimental/torch_xla2
configfile: pyproject.toml
plugins: benchmark-4.0.0, hydra-core-1.3.2, xdist-3.6.1
56 workers [0 items]

===================================== warnings summary =====================================
../../../../miniconda3/envs/xla2/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:335: 56 warnings
  /mnt/hanq/miniconda3/envs/xla2/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:335: UserWarning: Device capability of jax unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== 56 warnings in 9.02s ===================================
(xla2)
hanq@hanq-compile-2:/mnt/hanq/pytorch/xla/experimental/torch_xla2$`

which shows 0 test ran.

After this change:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================= short test summary info ==================================
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager__unsafe_masked_index_put_accumulate_cpu_int64 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_cov_cpu_int64 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager__unsafe_masked_index_put_accumulate_cpu_float32 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_cov_cpu_float32 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_gather_cpu_float32 - IndexError: Too many indices: 1-dimensional array indexed with 2 regular indices.
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_gather_cpu_int64 - IndexError: Too many indices: 1-dimensional array indexed with 2 regular indices.
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_nn_functional_multi_head_attention_forward_cpu_float32 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_scatter_cpu_int64 - AssertionError: Scalars are not close!
================ 8 failed, 1019 passed, 25 skipped, 100 warnings in 37.20s =================
(xla2) hanq@hanq-compile-2:/mnt/hanq/pytorch/xla/experimental/torch_xla2$ pytest test/test_ops.py

There are 8 extra test failures, presumably regressions introduced when this test was no longer running.

This PR reenable this test and mark the failing ones as skipped to stop further regression.

When we updated torch version to 2.6, one behavior of test_ops changed.

This line `instantiate_device_type_tests(TestOpInfo, globals(),
only_for='cpu')` now treats only_for argument as iteratable instead of
one string. so 'cpu' is split into ['c', 'p', 'u']; and this test did
not run.

Running `pytest test/test_ops.py` gives

```
(xla2) hanq@hanq-compile-2:/mnt/hanq/pytorch/xla/experimental/torch_xla2$ pytest test/test_ops.py
/mnt/hanq/miniconda3/envs/xla2/lib/python3.10/site-packages/pytest_benchmark/logger.py:46: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
  warner(PytestBenchmarkWarning(text))
=================================== test session starts ====================================
platform linux -- Python 3.10.14, pytest-8.3.2, pluggy-1.5.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /mnt/hanq/pytorch/xla/experimental/torch_xla2
configfile: pyproject.toml
plugins: benchmark-4.0.0, hydra-core-1.3.2, xdist-3.6.1
56 workers [0 items]

===================================== warnings summary =====================================
../../../../miniconda3/envs/xla2/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:335: 56 warnings
  /mnt/hanq/miniconda3/envs/xla2/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:335: UserWarning: Device capability of jax unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== 56 warnings in 9.02s ===================================
(xla2)
hanq@hanq-compile-2:/mnt/hanq/pytorch/xla/experimental/torch_xla2$`
```
which shows 0 test ran.

After this change:

```
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================= short test summary info ==================================
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager__unsafe_masked_index_put_accumulate_cpu_int64 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_cov_cpu_int64 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager__unsafe_masked_index_put_accumulate_cpu_float32 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_cov_cpu_float32 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_gather_cpu_float32 - IndexError: Too many indices: 1-dimensional array indexed with 2 regular indices.
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_gather_cpu_int64 - IndexError: Too many indices: 1-dimensional array indexed with 2 regular indices.
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_nn_functional_multi_head_attention_forward_cpu_float32 - AssertionError: Tensor-likes are not close!
FAILED test/test_ops.py::TestOpInfoCPU::test_reference_eager_scatter_cpu_int64 - AssertionError: Scalars are not close!
================ 8 failed, 1019 passed, 25 skipped, 100 warnings in 37.20s =================
(xla2) hanq@hanq-compile-2:/mnt/hanq/pytorch/xla/experimental/torch_xla2$ pytest test/test_ops.py
```
There are 8 extra test failures, presumably regressions introduced when
this test was no longer running.

This PR reenable this test and mark the failing ones as skipped to stop
further regression.
@qihqi qihqi requested a review from lsy323 December 23, 2024 19:45
@lsy323 lsy323 merged commit c385359 into master Dec 23, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants