Investigate DeepSpeed Inference

[Quentin-Anthony]

DeepSpeed wins most inference benchmarks I see. We should test their claims on neox models. EleutherAI spends a significant amount of compute running inference, so any improvement in inference performance would be high-impact. What I would like to see are:

1. Take baseline inference latency on a small/medium/large (160M, 6.9B, 20B) neox models with gpt-neox.
2. A summarization on how DeepSpeed's inference engine can benefit us (https://arxiv.org/abs/2207.00032, https://www.deepspeed.ai/tutorials/inference-tutorial/, https://github.com/microsoft/DeepSpeed/blob/master/docs/_tutorials/inference-tutorial.md)
3. Take new numbers on the above models with DeepSpeed inference, compare with baseline, write up a short README and include numbers + run instructions for DS inference.

[cr458]

would be more than happy to take this on, are there any resources available for these tests or is it up to me to find them?

[StellaAthena]

would be more than happy to take this on, are there any resources available for these tests or is it up to me to find them?

If by “resources” you mean computing resources, yes we can easily make GPUs available for testing this PR.

[cr458]

Hi Stella, thanks for your reply. Yup that's exactly what I meant, sorry about the poor wording. Great, in that case I'd love to take this on. Let me know how you'd like to arrange access to the GPUs and I'll get going!

[satpalsr]

+1

[StellaAthena]

@cr458 @satpalsr What's the current status of this?

[IshanMi]

Hey @Quentin-Anthony - I would love the chance to work on this!

[yang]

Some initial numbers are promising. With the current configs/125M.yml and text_generation.yml, I see some pretty consistent numbers of ~2.4s duration_seconds go down to ~1.4s, on a single-GPU node (A10G). Will share more numbers once I can get compute.