[LLM] Unify pipeline model with PretrainModelPipe

[ZHUI]

PR types

New features

PR changes

Others

Description

Unify pipeline model with PretrainModelPipe

import paddle
import paddle.distributed.fleet as fleet
from paddlenlp.transformers import AutoModelForCausalLMPipe, AutoModelForCausalLM

world_size = paddle.distributed.get_world_size()

pp_degree = 2
tp_degree = world_size/2

strategy = fleet.DistributedStrategy()
strategy.hybrid_configs = {
    "dp_degree": 1,
    "mp_degree": tp_degree,
    "pp_degree": pp_degree,
    "sharding_degree": 1,
}
fleet.init(is_collective=True, strategy=strategy)
hcg = fleet.get_hybrid_communicate_group()

if pp_degree > 1:
    model_class = AutoModelForCausalLMPipe
else:
    model_class = AutoModelForCausalLM

model_name_or_path = "facebook/llama-7b"
# model_name_or_path = "__internal_testing__/tiny-random-llama"
model = model_class.from_pretrained(
    model_name_or_path,
    tensor_parallel_degree=tp_degree,
    tensor_parallel_rank=hcg.get_model_parallel_rank(),
    tensor_parallel_output=False,
)

model.eval()

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Merging #7095 (d7015d6) into develop (d2524ab) will decrease coverage by 0.29%.
Report is 8 commits behind head on develop.
The diff coverage is 34.58%.

@@             Coverage Diff             @@
##           develop    #7095      +/-   ##
===========================================
- Coverage    59.84%   59.55%   -0.29%     
===========================================
  Files          557      563       +6     
  Lines        82150    82775     +625     
===========================================
+ Hits         49161    49299     +138     
- Misses       32989    33476     +487     

Files Changed	Coverage Δ
paddlenlp/transformers/model_utils.py	64.03% <19.51%> (-4.08%)	⬇️
paddlenlp/transformers/auto/modeling.py	77.94% <25.00%> (-2.63%)	⬇️
paddlenlp/transformers/llama/modeling_pp.py	24.48% <70.00%> (ø)
paddlenlp/transformers/__init__.py	100.00% <100.00%> (ø)
paddlenlp/transformers/gpt/__init__.py	100.00% <100.00%> (ø)
paddlenlp/transformers/gpt/configuration.py	100.00% <100.00%> (ø)
paddlenlp/transformers/gpt/modeling_pp.py	31.39% <100.00%> (ø)
paddlenlp/transformers/llama/__init__.py	100.00% <100.00%> (ø)
paddlenlp/transformers/llama/configuration.py	100.00% <100.00%> (ø)

... and 7 files with indirect coverage changes

[wawltor]

这里的vocab size的改动目的是什么

[wawltor]

同时改动vocab size之后会对后续热启word embedding有影响吗

[ZHUI]

适配TP，随机初始化的才改

[wawltor]

影响精度

[wawltor]

LGTM