v1.5.2

English Version

1. Support show log in text box of web-ui
2. Support share=True in web-ui, only need to set WEBUI_SHARE=1 in environment variable
3. Support deactivate all adapters
4. Support more SFT arguments
5. Add longlora/qalora script
6. Support custom models in web-ui
7. ModelScope SWIFT studio released: https://www.modelscope.cn/studios/damo/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary
8. Fix some bugs

---

中文版本

1. 支持在web-ui中直接显示日志
2. 支持share=True 仅需要在环境变量中设置WEBUI_SHARE=1
3. 支持失活所有adapters
4. 添加了更多SFT参数
5. 添加了longlora/qalora的训练脚本
6. web-ui支持了自己注册的自定义模型
7. SWFT魔搭创空间上线了: https://www.modelscope.cn/studios/damo/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary
8. 修复了一些bug

What's Changed

• fix chatglm3 template bug by @Jintao-Huang in #298
• Support studio by @tastelikefeet in #300
• fix text label by @tastelikefeet in #301
• fix a bug may cause module on gpu throws error by @tastelikefeet in #302
• fix_ziya_template_bug by @Jintao-Huang in #303

Full Changelog: v1.5.1...v1.5.2

v1.5.1

English version

New Features

1. Support dtype settings in LoRA
2. Support deactivated tuners offloading
3. Support deployment with OpenAI format restful API
4. Make LongLoRA supports the latest llama2 code

新feature

1. 支持LoRA设置dtype类型
2. 支持将不使用的tuners offloading到cpu和meta设备上
3. 支持OpenAI restful API方式的部署
4. LongLoRA支持最新的llama2代码

What's Changed

• update docs by @tastelikefeet in #269
• Update benchmark 0101 by @Jintao-Huang in #271
• update benchmark by @Jintao-Huang in #276
• Support dtype in lora by @tastelikefeet in #278
• Support deploy by @Jintao-Huang in #275
• update deploy client by @Jintao-Huang in #279
• Support offload by @tastelikefeet in #281
• fix tuner bug by @Jintao-Huang in #285
• Support lora_modules_to_save by @Jintao-Huang in #284
• update template by @Jintao-Huang in #286
• support ModuleToSave original module offloading by @tastelikefeet in #282
• Fix offload by @tastelikefeet in #288
• fix scedit bug by @tastelikefeet in #290
• fix bnb qwen bug by @Jintao-Huang in #289
• update readme by @Jintao-Huang in #291
• update readme by @Jintao-Huang in #292
• Fix/longlora by @tastelikefeet in #294
• support additional_trainable_parameters by @Jintao-Huang in #295
• fix webui by @tastelikefeet in #296
• fix trainer push to hub by @Jintao-Huang in #297

Full Changelog: v1.5.0...v1.5.1

v1.5.0

English Version

New features:

1. Support multi-line inference
2. Support multi node training
3. Add benchmarks
4. Support UI training, start by swift web-ui
5. Support VLLM inference
6. Support RLHF(DPO) training

New tuners:

1. SCEdit, adopted by TongYi Lab, uses lesser memory but produces better performance than LoRA, and can be used to replace ControlNet in a series of scenarios like Pose control/In-Painting/Out-Paining/Label-removing, etc.

New models:

1. SUS series models
2. Mixtral-MoE series models
3. deepseek series models
4. phi2-3b
5. cogagent-chat/cogagent-vqa
6. codegeex2-6b

New datasets:

Datasets used in RLHF:

1. hh-rlhf
2. stack-exchange-paired

中文版

SWIFT本月新版本已经发布！

新特性:

1. 支持多行输入推理
2. 支持多卡训练
3. 添加了模型训练的benchmarks
4. 支持界面训练和推理，通过swift web-ui开启
5. 支持VLLM推理
6. 支持RLHF(DPO)训练

新tuners:

SCEdit: 通义实验室自研的优秀U-Net微调框架，显存占用远小于LoRA，效果较LoRA更好，且可以替代实现ControlNet的效果，实现In-Painting/Out-Paining/标签去除/Pose控制等能力

新模型：

SUS系列模型
Mixtral-MoE系列模型
deepseek系列模型
phi2-3b
cogagent-chat/cogagent-vqa
codegeex2-6b

新数据集:

用于RLHF的数据集：
hh-rlhf
stack-exchange-paired

What's Changed

• update multi-line input (infer) by @Jintao-Huang in #196
• Fix model saving in new format by @tastelikefeet in #198
• Fix compatible error by @tastelikefeet in #201
• Fix bug 1206 by @Jintao-Huang in #202
• fix fp16 & full bug by @Jintao-Huang in #203
• Fix qwen-audio inference bug by @Jintao-Huang in #204
• Support multi node by @Jintao-Huang in #205
• fix typo bug by @Jintao-Huang in #206
• Support sus by @Jintao-Huang in #207
• Support cpu by @Jintao-Huang in #208
• Add Feat: Freeze Parameters, disable_tqdm by @Jintao-Huang in #210
• update dataset by @Jintao-Huang in #212
• Support lazy_tokenize, preprocess_num_proc by @Jintao-Huang in #211
• Support Mixtral MoE by @tastelikefeet in #217
• Add benchmark by @Jintao-Huang in #213
• support ui training by @tastelikefeet in #219
• Fix transformers 4.36 by @Jintao-Huang in #218
• Update mixtral-7b-moe by @Jintao-Huang in #221
• Compatible with peft>=0.7.0 by @tastelikefeet in #220
• fix dtype='fp16' sft bug by @Jintao-Huang in #227
• fix generation_config warning by @Jintao-Huang in #224
• Fix merge_lora & model_cache_dir bug by @Jintao-Huang in #229
• fix lazy_tokenize bug by @Jintao-Huang in #228
• Add inference UI and refactor machenism by @tastelikefeet in #230
• Support deepseek by @Jintao-Huang in #223
• relax version restriction by @tastelikefeet in #232
• fix bug 1218 by @Jintao-Huang in #235
• support deployment by @Jintao-Huang in #231
• update docs by @Jintao-Huang in #238
• Refactor some code by @tastelikefeet in #237
• fix typo bug by @Jintao-Huang in #239
• update readme & phi2-3b by @Jintao-Huang in #241
• Fix argument 1220 by @Jintao-Huang in #242
• Support CogAgent by @tastelikefeet in #243
• fix infer by @tastelikefeet in #244
• Support more peft tuners by @tastelikefeet in #245
• Fix copying additional files by @tastelikefeet in #247
• Add sft for codegeex2 by @tastelikefeet in #248
• fix issue #249 by @tastelikefeet in #250
• Feat/scedit by @jiangzeyinzi in #253
• Update 1228 by @Jintao-Huang in #254
• fix unicode error by @tastelikefeet in #259
• Update readme for SCEdit by @tastelikefeet in #258
• DPO by @tastelikefeet in #255
• update self-cognition by @Jintao-Huang in #261
• Fix/1229 by @tastelikefeet in #260
• fix trainer init by @tastelikefeet in #262
• fix bugs by @tastelikefeet in #263
• fix import by @tastelikefeet in #265
• Fix import by @tastelikefeet in #266
• update perf by @Jintao-Huang in #264
• fix bug by @tastelikefeet in #267
• Support win32 by @tastelikefeet in #268

Full Changelog: v1.4.0...v1.5.0

v1.4.0

English Version

New features:

1. Support for self-awareness fine-tuning.
2. Support for fine-tuning and inference of the AnimateDiff model in the AIGC direction.
3. Support for flash attention in more models: qwen series, qwen-vl series, llama series, openbuddy series, mistral series, yi series, ziya series, using the use_flash_attn parameter.
4. Support for multiple loras to take effect simultaneously.

New tuners:

1. NEFTune
2. ROME supports more models: chatglm

New models:

1. AnimateDiff
2. zephyr-7b-beta-chat, openbuddy-zephyr-7b-chat
3. qwen-1_8b, qwen-1_8b-chat, qwen-1_8b-chat-int4, qwen-1_8b-chat-int8
4. qwen-72b, qwen-72b-chat, qwen-72b-chat-int4, qwen-72b-chat-int8
5. qwen-audio, qwen-audio-chat
6. yi-34b-chat, codefuse-codellama-34b-chat
7. tongyi-finance-14b, tongyi-finance-14b-chat, tongyi-finance-14b-chat-int4
8. bluelm-7b, bluelm-7b-chat, bluelm-7b-32k, bluelm-7b-chat-32k

New datasets:

1. hc3-zh, hc3-en
2. codefuse-python-en, codefuse-eval-instruction-zh
3. aishell1-zh, aishell1-mini-zh

中文版

新特性:

1. 支持自我认知微调.
2. 支持AIGC方向的AnimateDiff模型的微调与推理.
3. 支持更多模型的flash attention: qwen series, qwen-vl series, llama series, openbuddy series, mistral series, yi series, ziya series. 使用use_flash_attn参数.
4. 支持多个lora同时生效

新tuners:

1. NEFTune
2. ROME支持更多模型: chatglm

新模型:

1. AnimateDiff
2. zephyr-7b-beta-chat, openbuddy-zephyr-7b-chat
3. qwen-1_8b, qwen-1_8b-chat, qwen-1_8b-chat-int4, qwen-1_8b-chat-int8
4. qwen-72b, qwen-72b-chat, qwen-72b-chat-int4, qwen-72b-chat-int8
5. qwen-audio, qwen-audio-chat
6. yi-34b-chat, codefuse-codellama-34b-chat
7. tongyi-finance-14b, tongyi-finance-14b-chat, tongyi-finance-14b-chat-int4
8. bluelm-7b, bluelm-7b-chat, bluelm-7b-32k, bluelm-7b-chat-32k

新数据集:

1. hc3-zh, hc3-en
2. codefuse-python-en, codefuse-eval-instruction-zh
3. aishell1-zh, aishell1-mini-zh

What's Changed

• Support Yi-6b sft by @tastelikefeet in #134
• fix CLI by @tastelikefeet in #135
• update readme by @tastelikefeet in #137
• Support xverse 65b sft by @tastelikefeet in #138
• Support bluelm by @Jintao-Huang in #140
• fix doc by @tastelikefeet in #143
• Add neftune by @tastelikefeet in #145
• Update sh by @Jintao-Huang in #144
• Add compatibility test and fix some problems with peft>=0.6.0 by @tastelikefeet in #146
• fix compatible with transformers>=4.35 by @Jintao-Huang in #148
• Update sh 1115 by @Jintao-Huang in #150
• Update doc by @tastelikefeet in #151
• support flash_attn by @Jintao-Huang in #152
• Fix bug: not work on peft<=0.5.0 by @tastelikefeet in #155
• fix register model bug by @Jintao-Huang in #154
• Support tongyi finance 14b by @Jintao-Huang in #157
• add check_model args and fix check_dataset by @Jintao-Huang in #159
• fix load_from_ckpt_dir bug by @Jintao-Huang in #161
• Update arguments by @Jintao-Huang in #162
• new feature: save_infer_result_to_jsonl by @Jintao-Huang in #163
• Feat 1121 by @Jintao-Huang in #165
• update readme and fix bug by @Jintao-Huang in #167
• Add cli merge lora by @Jintao-Huang in #168
• update code by @Jintao-Huang in #169
• support yi-34b-chat by @Jintao-Huang in #164
• Add animate diff by @tastelikefeet in #174
• update readme by @Jintao-Huang in #175
• Refine LoRA to peft by @tastelikefeet in #176
• support qwem-72b qwen-1_8b qwen-audio by @Jintao-Huang in #180
• Update wechat by @Jintao-Huang in #186
• Fix the slow inference speed bug in qwen AutoGPTQ by @Jintao-Huang in #187
• Support self cognition by @Jintao-Huang in #188
• update dataset model by @Jintao-Huang in #190

Full Changelog: v1.3.0...v1.4.0

v1.3.0 Release

English Version

New Features:

1. Serving supported: LoRA and full-parameter training models are supported in vllm/chatglm.cpp/xinference deployment, check the documentation for details by make docs or docs/source/GetStarted/Deployment.md file.
2. Support training and inference with CLI and inference with Web-UI.

New Adapters:

• QALoRA
• Long-LoRA
• ROME

New Models:

1. xverse-65b
2. yi-6b
3. ziya2-13b
4. ziya2-13b-chat
5. mistral-7b
6. openbuddy-mistral-7b-chat
7. mistral-7b-chat
8. chatglm3-6b-base
9. chatglm3-6b
10. chatglm3-6b-32k

New Quantized Models:

1. qwen-7b-chat-int4
2. qwen-14b-chat-int4
3. qwen-vl-chat-int4
4. baichuan2-7b-chat-int4
5. baichuan2-13b-chat-int4
6. qwen-7b-chat-int8
7. qwen-14b-chat-int8

---

中文版
新功能:

1. 支持部署：全参数训练及LoRA训练支持以vllm/chatglm.cpp/xinference方式进行部署，可以通过make docs生成官方文档或查看docs/source/GetStarted/Deployment.md文件
2. 支持CLI方式运行训练和推理，同时支持以Web-UI方式运行推理

新的Adapters：

• QALoRA
• Long-LoRA
• ROME

支持训练和推理的新模型:

1. xverse-65b
2. yi-6b
3. ziya2-13b
4. ziya2-13b-chat
5. mistral-7b
6. openbuddy-mistral-7b-chat
7. mistral-7b-chat
8. chatglm3-6b-base
9. chatglm3-6b
10. chatglm3-6b-32k

支持训练和推理的新量化模型:

1. qwen-7b-chat-int4
2. qwen-14b-chat-int4
3. qwen-vl-chat-int4
4. baichuan2-7b-chat-int4
5. baichuan2-13b-chat-int4
6. qwen-7b-chat-int8
7. qwen-14b-chat-int8

---

Feature Commits

• add lint script by @tastelikefeet in #94
• add document by @Jintao-Huang in #103
• Update framework.txt by @zzclynn in #105
• Feat/deepspeed by @Jintao-Huang in #39
• update sh by @Jintao-Huang in #106
• Add baichuan2 13b sh by @Jintao-Huang in #108
• Support mistral 7b by @Jintao-Huang in #112
• Support int4 by @Jintao-Huang in #116
• Support qwen int8 by @Jintao-Huang in #117
• support qa_lora by @tastelikefeet in #104
• Add longlora for llama by @tastelikefeet in #115
• Support ROME by @tastelikefeet in #121
• Feat 1018 by @Jintao-Huang in #119
• Add script for ROME by @tastelikefeet in #123
• Update doc by @tastelikefeet in #125
• Feat 1028 by @Jintao-Huang in #122
• update web ui and swift cli by @Jintao-Huang in #126
• ResTuning readme by @jiangzeyinzi in #129
• Update doc by @tastelikefeet in #131

Bug Fix:

• Fix qwen bug by @Jintao-Huang in #98
• fix ci by @tastelikefeet in #109
• fix resume from checkpointing bug by @Jintao-Huang in #110
• fix openbuddy template by @Jintao-Huang in #113
• update ziya2 by @Jintao-Huang in #114
• Fix bug in ROME by @tastelikefeet in #128
• fix rome by @tastelikefeet in #133

New Contributors

• @zzclynn made their first contribution in #105

Full Changelog: v1.2.0...v1.3.0

v1.1.1 release

Features:

1. Add make docs command to build docs
2. Add notebook examples for stable diffusion model
3. Fix some bugs

v1.1.0

v1.1.0

What's Changed

Features

• Support qwen sft by @tastelikefeet
• Add quantization by @Jintao-Huang
• Add dataset and README_CN by @Jintao-Huang
• Add polylm-13b, openbuddy-llama2-13b by @Jintao-Huang
• Add models and update sh, readme by @Jintao-Huang
• llm add chat by @Jintao-Huang
• llm support multimodal by @Jintao-Huang
• Feat/multi round chat by @Jintao-Huang
• Add prompt feature by @jiangzeyinzi
• Add openbuddy llama2 by @Jintao-Huang
• Add feat: only save model by @Jintao-Huang
• Add baichuan2 by @Jintao-Huang
• Add bloom by @Jintao-Huang
• Add internlm by @Jintao-Huang
• Support restuning/side/lora-embedding/qlora, etc by @tastelikefeet

Improvements

• Update README.md by @yingdachen
• Fix sample by @tastelikefeet
• Add comment by @Jintao-Huang
• Update sh2 by @Jintao-Huang

Fixes

• Fix not saving the last ckpt bug by @Jintao-Huang
• Fix ddp rank0 memory bug by @Jintao-Huang
• Fix template bug by @Jintao-Huang
• Fix template bug2 by @Jintao-Huang
• Fix ddp bug by @Jintao-Huang
• Fix ddp+graidient_checkpointing bug by @Jintao-Huang
• Fix httperror by @weedwardzhao1

New Contributors

• @yingdachen made their first contribution in PR #2
• @tastelikefeet made their first contribution in PR #3
• @Jintao-Huang made their first contribution in PR #7
• @jiangzeyinzi made their first contribution in PR #38
• @weedwardzhao1 made their first contribution in PR #71

Full Changelog: https://github.com/modelscope/swift/commits/v1.1.0