I have a question about multi-GPU inference

[gaoxt1983]

In the inference tutorial: https://www.deepspeed.ai/tutorials/inference-tutorial/ , for this example:

# Filename: gpt-neo-2.7b-generation.py
import os
import deepspeed
import torch
from transformers import pipeline

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))
generator = pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B',
                     device=local_rank)



generator.model = deepspeed.init_inference(generator.model,
                                           mp_size=world_size,
                                           dtype=torch.float,
                                           replace_with_kernel_inject=True)

string = generator("DeepSpeed is", do_sample=True, min_length=50)
if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
    print(string)

I want to know:

1. if I'm using deepspeed --num_gpus 2 gpt-neo-2.7b-generation.py, the "generator" statement runs once or twice?
2. should I do something necessary for different rank of machine?

[zjlxgxz]

it seems it run on every process (gpu). By the way, DeepSpeed inherently needs to sync across GPUs, if the inputs are different across GPU, it seems hang forever with 100% GPU utilization (which is kind of confusing).

[zjlxgxz]

ps: It seems DS's tensor parallelism needs the same input on every device, so weight tensors on different GPUs can get the same input data for sub-matrix calculation at the same time.