LLM

[zhangjun]

repo链接

https://github.com/THUDM/ChatGLM-6B
https://github.com/mymusise/ChatGLM-Tuning
https://github.com/LianjiaTech/BELLE

LLM量化

https://zhuanlan.zhihu.com/p/616969812

• SmoothQuant
• Outlier Suppression
  • Outlier Suppression
  • Outlier Suppression+
• AWQ
  基于激活与参数大小的缩放保护
• LLM.int8
  • LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  • https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/
  • https://github.com/timdettmers/bitsandbytes
• GPTQ
  • GPTQ: 模型量化，穷鬼救星
    不使用贪心算法，对W进行优化时，固定位置挑选Q
    W不同列间权重更新互相独立，使用批处理更新（Lazy Batch更新，group_size参数控制）
    数值稳定性优化
  • 4-bit LLM Quantization with GPTQ
• ZeroQuant
  • Ds-inference Int8 support through ZeroQuant technology deepspeedai/DeepSpeed#2217
  • ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
• LUT-GEMM
  • LUT-GEMM: 基于lut的量化矩阵乘法在大规模生成语言模型中的高效推理
• SparseGPT
• weight only
  • Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production
  • TVM weight-only
  • cutlass_fpA_intB_gemm

fine-tune

• PEFT
• QLora

LLM papers

Awesome-LLM-System-Papers
SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification
https://flexflow.ai/specInfer/
https://www.gabrieleoliaro.com/publication/expertflow/expertflow.pdf

[zhangjun]

LLM Inference Optimization

https://lilianweng.github.io/posts/2023-01-10-inference-optimization/
https://www.nebuly.com/blog/full-stack-optimization-of-transformer-inference-a-survey-part-1
Full Stack Optimization of Transformer Inference: a Survey

continous batching

continuous batching llm inference

[zhangjun]

TensorRT & Triton

Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server

[zhangjun]

大模型分布式推理

推理方案

	<=百亿	百亿～千亿	>千亿
资源	单机多卡	单机多卡	多机多卡
并行策略	张量并行	张量并行	张量+流水线并行
通讯	单机all reduce	单机all reduce	单机all reduce+多机send/recv

模型切分策略

[zhangjun]

LLM应用

LangChain

[zhangjun]

Flash Attention

Flash Attention

• parallelism
  parallelize over batch_size and num of heads
  flash attention2 - long sequences(small batch size or num of heads), parallelize over sequence length dimension
• better work partition
  reduce the amount of synchronization and communication between different warps
  FlashAttention splits K and V across 4 warps while keeping Q accessible by all warps.
  FlashAttention-2 splits Q across 4 warps while keeping K and V accessible by all warps.
  
• supported head dimensions up to 256 and MQA
  GQA、MQA

Flash Attention2 优化

Flash Attention2优化点详解

• Fused kernel与矩阵分块
• Causal Masking
• Non-Matmul 计算优化
• 流水编排与异步加载和Double Buffer
• Layout Swizzle

[zhangjun]

TensorRT-LLM

streaming of tokens, in-flight batching, paged-attention, quantization

[zhangjun]

MeMo

megatron_lm_encoder_decoder_model.py

[zhangjun]

Smooth-Quant int8 quantization achieves 2.46x faster latency compared to fp16 mode. (OPT-13B on A100 80GB)

[zhangjun]

Fine Tune

Prompt Series

P-Tuning V2
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
The Power of Scale for Parameter-Efficient Prompt Tuning
P-Tuning GPT Understands, Too

Lora Series

LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS

SFT

大模型SFT微调指令数据的生成