[small] use torch.int for autotuning cache

[ngimel]

For stupid reasons, ops on int8 are 3 times slower than on int, and for another set of stupid reasons we are not using cudaMemset for zero_, so using int8 buffer in do_bench makes it slow.

[ngimel]

Don't think perf test failures are related, 170us that cache clearing kernel takes (on A100, even longer on other devices) is still more than enough to cover any launch overheads, so reported time shouldn't change.

[ptillet]

Let me relaunch

[ngimel]

Locally some perf tests are failing for me both with and without this change

[ptillet]

Yeah that's weird. Sometimes the CI does that. Let me restart the CI server -- this has always solved these sort of flaky perf tests :)

[ptillet]

Since it failed even after a CI reboot while a re-run of the tests on the master passed (https://github.com/openai/triton/actions/runs/3392284659), I am guessing that this PR for some mysterious reasons at least modifies the baseline. Might be related to a discussion I had with @Jokeren about how flushing the L2 cache may affect the performance of matmuls. We haven't got to the bottom of it yet 😅

[Jokeren]

Got you. The issue is that flushing the L2 cache before running a matmul can promote the performance a bit.

[ngimel]

But this PR flushes L2 cache in the same way it used to (the size of cache is the same), just using a faster kernel to do so.

[ptillet]

Yeah, that's weird. In practice, I guess potentially anything that modifies do_bench could require us to recalibrate the baseline perf numbers in our regression tests. How important is this PR for torch-inductor? If it's not urgent, is it ok for me to keep this PR open and delay its merging until we have validated Triton-MLIR to have the same performance as the current baseline with the current do_bench -- out of an abundance of caution ?

[ngimel]

We'd like to have it in as it noticeably reduces our startup latency, but we can delay for couple weeks. Alternatively, what about fast_flush arg for do_bench?

[ptillet]

Yep, good idea! let's add an extra argument that preserves the old behavior by default

[ngimel]

Hm failure is really weird, now default behavior is strictly unchanged

[ptillet]

I think it's fine to change the baseline for "test_performance.py::test_elementwise[1048576] FAILED" to 0.52 instead of 0.53. Obviously an artifact of having extra control flow in the loop, but not a big deal since it's elementwise

[ngimel]

lol and now observed is 0.54 and thus failing tests, this is too delicate!

[ngimel]

btw there's no control flow in the loop, cache is created once before measurement loop.

[ptillet]

Thanks! And sorry for the friction :)

[ngimel]

Thanks!

[eellison]

For stupid reasons, ops on int8 are 3 times slower than on int

Is this because of pytorch casting from int8 -> int32 -> int8 ? I would have thought zero_ would be bandwidth bound anyway.