Add tests in CI

[msaroufim]

This just runs all the tests in test on a T4 machine notable changes

• Make sure code still works if eleuther eval harness is not installed
• Remove useless tests like test install since we are testing whole suite
• Some notable skip tests that will require more work to enable by default
• Requirements.txt and dev-requirements.txt

Let's merge this ASAP so we can also start doing nightly releases with this #49

Locally

(ao) ubuntu@ip-172-31-36-68:~/ao$ pytest test/
========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.10.13, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/ubuntu/ao
collected 21 items                                                                                                                                                                                                                       

test/dtypes/test_nf4.py ....s.s..                                                                                                                                                                                                  [ 42%]
test/dtypes/test_uint4.py sss                                                                                                                                                                                                      [ 57%]
test/quantization/test_quant_api.py .sss                                                                                                                                                                                           [ 76%]
test/sparsity/test_wanda.py .....                                                                                                                                                                                                  [100%]

There's still some todos in a later PR

• Some tests assume the GPT checkpoints are locally available
• Some tests assume that you're using uint4 but that's not available expect in pytorch nightlies
• There's one test with a legitimate accuracy error
• There's a lot warning spew

============================================================================================================ warnings summary ============================================================================================================
test/dtypes/test_nf4.py::TestNF4Linear::test_to_copy
  /home/ubuntu/ao/test/dtypes/test_nf4.py:171: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(inpt_tensor, inpt_tensor_bfloat16, atol=0.13, rtol=0.13)

test/dtypes/test_nf4.py::TestNF4Linear::test_to_copy
  /home/ubuntu/ao/test/dtypes/test_nf4.py:177: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(inpt_tensor, inpt_tensor_bfloat16, atol=0.13, rtol=0.13)

test/quantization/test_quant_api.py::TestQuantFlow::test_dynamic_quant_gpu_singleline
test/quantization/test_quant_api.py::TestQuantFlow::test_dynamic_quant_gpu_singleline
  /opt/conda/envs/ao/lib/python3.10/site-packages/torch/ao/quantization/utils.py:339: UserWarning: must run observer before calling calculate_qparams. Returning default values.
    warnings.warn(

test/sparsity/test_wanda.py::TestWandaSparsifier::test_one_layer_mlp_2x4
  /home/ubuntu/ao/torchao/sparsity/wanda.py:42: UserWarning: WandaSparsifier got semi_structured_bock_size=4, sparsity_level fixed to 50% (2:4) sparsity
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================================================== 13 passed, 8 skipped, 5 warnings in 24.27s ===============================================================================================
(ao) ubuntu@ip-172-31-36-68:~/ao$ 

[jerryzh168]

should this live under if branch as well?

[msaroufim]

I would vote for no, because the eval wrappers are correctly skipped if eleuther harness is not available and I'd rather we seperate algorithm and correctness checks. In some sense though you're right in that this file needs better tests. My goal right now was to make existing tests green so I'd rather we revisit these changes

[cpuhrsch]

Agreed, we shouldn't need accuracy checks to run inference, so we shouldn't need lm_eval to use the quantizer.

[jerryzh168]

right now GPTQQuantizer has dependency on lm-eval though, it's not for accuracy eval, it's used to gather the inputs from tasks. not sure how we want to remove the dependency there