diff --git a/.gitignore b/.gitignore index 41e0d4be..7466f65e 100644 --- a/.gitignore +++ b/.gitignore @@ -27,8 +27,9 @@ datasets qqp glm_large_qqp_pytorch wandb +clip_benchmark_datasets examples/AltCLIP/clip_benchmark_datasets examples/glm_pretrain/data.lazy examples/glm_pretrain/examples/glm_pretrain/data.lazy examples/vit_cifar100/cifar100 -examples/vit_cifar100/data \ No newline at end of file +examples/vit_cifar100/data diff --git a/README.md b/README.md index 7dee911d..d48f1be3 100644 --- a/README.md +++ b/README.md @@ -15,11 +15,12 @@ FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensibl * These models can be applied to (Chinese/English) Text, for tasks like text classification, information extraction, question answering, summarization, and text generation. -* FlagAI is backed by the three most popular data/model parallel libraries — [PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)/[BMTrain](https://github.com/OpenBMB/BMTrain) — with seamless integration between them. Users can parallel their training/testing process with less than ten lines of code. +* FlagAI is backed by the four most popular data/model parallel libraries — [PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)/[BMTrain](https://github.com/OpenBMB/BMTrain) — with seamless integration between them. Users can parallel their training/testing process with less than ten lines of code. The code is partially based on [GLM](https://github.com/THUDM/GLM), [Transformers](https://github.com/huggingface/transformers)，[timm](https://github.com/rwightman/pytorch-image-models) and [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/Megatron-LM). ## News +- [2 Mar 2023] release v1.6.1, Support Galactica model [#234](https://github.com/FlagAI-Open/FlagAI/pull/234); BMInf, a low-resource inference package [#238](https://github.com/FlagAI-Open/FlagAI/pull/238), and examples for p-tuning [#227](https://github.com/FlagAI-Open/FlagAI/pull/238) - [12 Jan 2023] release v1.6.0, support a new parallel lib called [**BMTrain**](https://github.com/OpenBMB/BMTrain) and integate [**Flash Attention**](https://github.com/HazyResearch/flash-attention) to speedup training of Bert and Vit models, examples in [FlashAttentionBERT](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/bert_title_generation_english/train_flash_atten.py) and [FlashAttentionViT](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/vit_cifar100/train_single_gpu_flash_atten.py). Also add the contrastive search based text generation method [**SimCTG**](https://github.com/yxuansu/SimCTG) and DreamBooth finetuning based on AltDiffusion, examples in [AltDiffusionNaruto](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/AltDiffusion/dreambooth.py). - [28 Nov 2022] release v1.5.0, support 1.1B [**EVA-CLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/EVA_CLIP) and [ALM: A large Arabic Language Model based on GLM], examples in [**ALM**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/ALM) - [10 Nov 2022] release v1.4.0, support [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://arxiv.org/abs/2211.06679v1), examples in [**AltCLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP) and [**AltDiffusion**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltDiffusion) @@ -259,6 +260,6 @@ The majority of FlagAI is licensed under the [Apache 2.0 license](LICENSE), howe ### ↳ Star History
-[![Star History Chart](https://api.star-history.com/svg?repos=FlagAI-Open/FlagAI&type=Date)](https://star-history.com/#baaivision/EVA&Date) +![Star History Chart](https://api.star-history.com/svg?repos=FlagAI-Open/FlagAI&type=Date)]
diff --git a/README_zh.md b/README_zh.md index 2ab9eff9..2ed999f8 100644 --- a/README_zh.md +++ b/README_zh.md @@ -15,12 +15,13 @@ * 这些模型可以应用于文本，用于文本分类、信息提取、问答、摘要、文本生成等任务，尤其是中文。 -* 飞智由三个最流行的数据/模型并行库（[PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)/[BMTrain](https://github.com/OpenBMB/BMTrain)）提供支持，它们之间实现了无缝集成。 你可以用不到十行代码来并行你的训练/测试过程。 +* 飞智由四个最流行的数据/模型并行库（[PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)/[BMTrain](https://github.com/OpenBMB/BMTrain)）提供支持，它们之间实现了无缝集成。 你可以用不到十行代码来并行你的训练/测试过程。 本项目的部分代码基于[GLM](https://github.com/THUDM/GLM)，[Transformers](https://github.com/huggingface/transformers)，[timm](https://github.com/rwightman/pytorch-image-models) 和 [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/Megatron-LM). ## 动态 +- [2 Mar 2023] 支持v1.6.1版本, 增加Galactica模型 [#234](https://github.com/FlagAI-Open/FlagAI/pull/234), 大模型推理的低资源工具包BMInf [#238](https://github.com/FlagAI-Open/FlagAI/pull/238), 以及P-tuning样例 [#227](https://github.com/FlagAI-Open/FlagAI/pull/238) - [12 Jan 2023] 发布v1.6.0版本, 新增支持并行训练库 [**BMTrain**](https://github.com/OpenBMB/BMTrain) 以及集成 [**Flash Attention**](https://github.com/HazyResearch/flash-attention) 到 Bert 和 Vit 模型提速端到端训练, 示例见 [FlashAttentionBERT](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/bert_title_generation_english/train_flash_atten.py)和 [FlashAttentionViT](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/vit_cifar100/train_single_gpu_flash_atten.py). 同时增加了基于对比搜索的文本生成方法 [**SimCTG**](https://github.com/yxuansu/SimCTG) 以及基于 AltDiffusion 进行 DreamBooth 个性化微调, 示例见 [AltDiffusionNaruto](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/AltDiffusion/dreambooth.py). - [28 Nov 2022] 发布v1.5.0版本, 支持1.1B参数的 [**EVA-CLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/EVA_CLIP) 以及[ALM: 基于GLM的阿拉伯语大模型], 示例见[**ALM**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/ALM) - [10 Nov 2022] 发布v1.4.0版本, 支持[AltCLIP: 更改CLIP中的语言编码器以扩展语言功能](https://arxiv.org/abs/2211.06679v1), 示例见[**AltCLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP)以及[**AltDiffusion**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltDiffusion) diff --git a/doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md b/doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md index 9be5a623..763c345c 100644 --- a/doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md +++ b/doc_zh/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md @@ -24,7 +24,7 @@ ### 1. 数据加载 样例数据位于 /examples/bert_title_generation/data/ -需要在 ```trianer.py```文件中定义数据读取过程，例如： +需要在 ```trainer.py```文件中定义数据读取过程，例如： ```python def read_file(): src = [] diff --git a/doc_zh/TUTORIAL_21_OPTIMIZER.md b/doc_zh/TUTORIAL_21_OPTIMIZER.md new file mode 100644 index 00000000..d1527b59 --- /dev/null +++ b/doc_zh/TUTORIAL_21_OPTIMIZER.md @@ -0,0 +1,54 @@ +# 如何使用优化器 + +## 优化器是什么? +在机器学习和深度学习的语境下， +优化器（Optimizer）是指用于更新模型参数的算法或方法，以便最小化预测输出和实际输出之间的误差。 + +优化器的目标是找到最优的参数组合，以在给定任务上获得最佳性能。 +这个过程通常在机器学习模型的训练阶段执行。 + +优化器通过计算损失函数相对于模型参数的梯度，并使用这些信息来更新参数，以减少损失。 +有多种可用的优化算法，例如随机梯度下降（SGD）、Adagrad、Adam、RMSprop等，每种算法都有其优点和缺点。 + +优化器的选择取决于特定问题、数据集的大小、模型的复杂性和其他因素。 +一个好的优化器可以显著提高模型的训练速度和准确性。 + + + + +## 加载优化器 + +### 依赖 +#### adan +``` +python3 -m pip install git+https://github.com/sail-sg/Adan.git +``` +#### lion +``` +$ pip install lion-pytorch +``` +#### lamb +``` +$ pip install torch_optimizer +``` +#### 例子 +```python +>>> # currently FlagAI support adam, adamw, lion, adan, adafactor and lamb, which can be defined by setting optimizer_type when defining Trainer +>>> trainer = Trainer(env_type='pytorch', +>>> epochs=1, +>>> batch_size=2, +>>> eval_interval=100, +>>> log_interval=10, +>>> experiment_name='glm_large_bmtrain', +>>> pytorch_device='cuda', +>>> load_dir=None, +>>> lr=1e-4, +>>> num_gpus = 1, +>>> weight_decay=1e-2, +>>> save_interval=1000, +>>> hostfile='./hostfile', +>>> training_script=__file__, +>>> deepspeed_config='./deepspeed.json', +>>> optimizer_type='lion') #load optimizer +``` + diff --git a/doc_zh/TUTORIAL_3_MODEL.md b/doc_zh/TUTORIAL_3_MODEL.md index 407379d3..e5ca27de 100644 --- a/doc_zh/TUTORIAL_3_MODEL.md +++ b/doc_zh/TUTORIAL_3_MODEL.md @@ -20,7 +20,7 @@ ## From_pretrain `From_pretrain` 函数用于加载模型。同一个模型结构的模型可以用同一个class进行加载，比如`BERT-base-ch` 和`Roberta-base-ch`模型都能用`BertModel`这个`Class`进行加载。`From_pretrain`为了数据/模型并行的模型加载进行了特定优化，避免重复下载导致的资源浪费。 -通过调用`ClassName.from_pretrian()`来进行加载. +通过调用`ClassName.from_pretrain()`来进行加载. ### 从modelhub加载 现在我们支持从modelhub中下载[常用模型](#所有支持模型)，可以直接通过`from_pretrain`下载模型配置文件`config.json`，模型权重`pytorch_model.bin`，以及字典文件`vocab.txt`。例子： ```python diff --git a/docs/TUTORIAL_21_OPTIMIZER.md b/docs/TUTORIAL_21_OPTIMIZER.md new file mode 100644 index 00000000..b77db929 --- /dev/null +++ b/docs/TUTORIAL_21_OPTIMIZER.md @@ -0,0 +1,57 @@ +# How to use Optimizer + +## What is Optimizer? +In the context of machine learning and deep learning, +an optimizer is an algorithm or method used to update the parameters of a model in order to minimize the error between the predicted output and the actual output. + +The goal of an optimizer is to find the optimal set of parameters that can achieve the best performance on a given task. +This process is typically performed during the training phase of a machine learning model. + +Optimizers work by computing the gradients of the loss function with respect to the model parameters, +and using this information to update the parameters in the direction that reduces the loss. +There are various optimization algorithms available, +such as stochastic gradient descent (SGD), Adagrad, Adam, RMSprop, and more, each with their own advantages and disadvantages. + +The choice of optimizer depends on the specific problem, the size of the dataset, +the complexity of the model, and other factors. +A good optimizer can significantly improve the training speed and accuracy of a model. + + + + +## Loading optimizer + +### dependencies +#### adan +``` +python3 -m pip install git+https://github.com/sail-sg/Adan.git +``` +#### lion +``` +$ pip install lion-pytorch +``` +#### lamb +``` +$ pip install torch_optimizer +``` +#### example +```python +>>> # currently FlagAI support adam, adamw, lion, adan, adafactor and lamb, which can be defined by setting optimizer_type when defining Trainer +>>> trainer = Trainer(env_type='pytorch', +>>> epochs=1, +>>> batch_size=2, +>>> eval_interval=100, +>>> log_interval=10, +>>> experiment_name='glm_large_bmtrain', +>>> pytorch_device='cuda', +>>> load_dir=None, +>>> lr=1e-4, +>>> num_gpus = 1, +>>> weight_decay=1e-2, +>>> save_interval=1000, +>>> hostfile='./hostfile', +>>> training_script=__file__, +>>> deepspeed_config='./deepspeed.json', +>>> optimizer_type='lion') #load optimizer +``` + diff --git a/docs/TUTORIAL_3_MODEL.md b/docs/TUTORIAL_3_MODEL.md index 60d3c3a7..adb2b88a 100644 --- a/docs/TUTORIAL_3_MODEL.md +++ b/docs/TUTORIAL_3_MODEL.md @@ -25,7 +25,7 @@ All supported models now support the three most common model types [encoder, dec ### load model from modelhub -By calling `ClassName.from_pretrian()` to load following [supported models](#all-supported-models), it will automatically download the model configuration file `config.json`, model weights `pytorch_model.bin`, and dictionary files `vocab .txt`. +By calling `ClassName.from_pretrain()` to load following [supported models](#all-supported-models), it will automatically download the model configuration file `config.json`, model weights `pytorch_model.bin`, and dictionary files `vocab .txt`. ```python >>> # Downloading GLM-large-ch from modelhub diff --git a/examples/bert_title_generation_english/generate.py b/examples/bert_title_generation_english/generate.py index 1124d16d..fdfa2f41 100644 --- a/examples/bert_title_generation_english/generate.py +++ b/examples/bert_title_generation_english/generate.py @@ -7,7 +7,7 @@ device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -model_dir = "../state_dict/" +model_dir = "./checkpoints/" # Note "./checkpoints_seq2seq/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_seq2seq. model_save_path = "./checkpoints_seq2seq/7079/mp_rank_00_model_states.pt" diff --git a/examples/bert_title_generation_english/train.py b/examples/bert_title_generation_english/train.py index a7f3423e..f22c6609 100644 --- a/examples/bert_title_generation_english/train.py +++ b/examples/bert_title_generation_english/train.py @@ -1,7 +1,6 @@ # Copyright © 2022 BAAI. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License") -import sys import os import torch from torch.utils.data import Dataset diff --git a/examples/bminf_generate/README.md b/examples/bminf_generate/README.md new file mode 100644 index 00000000..4240b9df --- /dev/null +++ b/examples/bminf_generate/README.md @@ -0,0 +1,45 @@ + +# BMInf + +## 简介/Overview + +BMInf is a low-resource inference package for large-scale pretrained language models. + +BMInf supports running models with more than 10 billion parameters on a single NVIDIA GTX 1060 GPU in its minimum requirements. Running with better GPUs leads to better performance. In cases where the GPU memory supports the large model inference (such as V100 or A100), BMInf still has a significant performance improvement over the existing PyTorch implementation. + +BMInf Github Repository address: https://github.com/OpenBMB/BMInf + +BMInf (Big Model Inference) 是一个用于大规模预训练语言模型（pretrained language models, PLM）推理阶段的低资源工具包。 + +BMInf最低支持在NVIDIA GTX 1060单卡运行百亿大模型。在此基础上，使用更好的gpu运行会有更好的性能。在显存支持进行大模型推理的情况下（如V100或A100显卡），BMInf的实现较现有PyTorch版本仍有较大性能提升。 + +BMInf 仓库地址：https://github.com/OpenBMB/BMInf + +## 应用/Application + +在模型加载参数之后，使用如下代码来用BMInf转换模型 + +```Python +with torch.cuda.device(0): + model = bminf.wrapper(model, quantization=False, memory_limit=20 << 30) +``` +The `quantization` parameter represents whether to use the model quantization technique, but if it is a generated class model, it needs to be set to `False`. + +You can use the `memory_limit` parameter to set the maximum available storage, the unit is Mb. + +`quantization`参数代表是否使用了模型量化的技巧，但如果是生成类模型，则需要设置成`False` + +可以用`memory_limit`参数设置最大的可用存储，单位为Mb + +如果`bminf.wrapper`不能很好的适配你的模型，你可以用以下的方法来进行手动适配。 + +* 将 `torch.nn.ModuleList` 替换为 `bminf.TransformerBlockList`. +```python +module_list = bminf.TransformerBlockList([ +], [CUDA_DEVICE_INDEX]) +``` + +* 将 `torch.nn.Linear` 替换为 `bminf.QuantizedLinear`. +```python +linear = bminf.QuantizedLinear(torch.nn.Linear(...)) +``` \ No newline at end of file diff --git a/examples/bminf_generate/cpm1_generate.py b/examples/bminf_generate/cpm1_generate.py new file mode 100644 index 00000000..81bb74e4 --- /dev/null +++ b/examples/bminf_generate/cpm1_generate.py @@ -0,0 +1,35 @@ +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +import bminf +import time + + +if __name__ == '__main__': + + text = '''默写古诗: + 白日依山尽，黄河入海流。 + 床前明月光，''' + + loader = AutoLoader(task_name="lm", + model_name="CPM-large-ch", + model_dir="./checkpoints", + device="cpu") + + model = loader.get_model() + time_start=time.time() + with torch.cuda.device(0): + model = bminf.wrapper(model, quantization=False, memory_limit=20 << 30) + tokenizer = loader.get_tokenizer() + + predictor = Predictor(model=model, + tokenizer=tokenizer, + ) + + out = predictor.predict_generate_randomsample(text, + top_p=0.9, + out_max_length=50) + time_end=time.time() + print('time cost',time_end-time_start,'s') + + print(out) diff --git a/examples/bminf_generate/galactica_6.7b_generate.py b/examples/bminf_generate/galactica_6.7b_generate.py new file mode 100644 index 00000000..29e8d8df --- /dev/null +++ b/examples/bminf_generate/galactica_6.7b_generate.py @@ -0,0 +1,37 @@ + +from flagai.model.predictor.predictor import Predictor +from flagai.auto_model.auto_loader import AutoLoader +import torch +import bminf +import time +device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + + +loader = AutoLoader(task_name="lm", + model_name="galactica-6.7b-en", + model_dir="./checkpoints/") + +model = loader.get_model() +with torch.cuda.device(0): + model = bminf.wrapper(model, quantization=False, memory_limit=20 << 30) +model.to(device) +model.eval() +tokenizer = loader.get_tokenizer() +predictor = Predictor(model, tokenizer) +print("model loaded") +time_start=time.time() + +text = "Please write a abstract about the computer vision. \n" +out = predictor.predict_generate_randomsample(text, + out_max_length=700, + top_k=50, + repetition_penalty=1.2, + temperature=0.7 + ) + +time_end=time.time() +print('time cost',time_end-time_start,'s') +print(out) + + + diff --git a/examples/bminf_generate/glm_generate.py b/examples/bminf_generate/glm_generate.py new file mode 100644 index 00000000..ddf3f730 --- /dev/null +++ b/examples/bminf_generate/glm_generate.py @@ -0,0 +1,20 @@ +from flagai.model.glm_model import GLMModel +from flagai.data.tokenizer import Tokenizer +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +import torch +import bminf + +model_name = 'GLM-10b-ch' +loader = AutoLoader("lm", 'GLM-10b-ch', model_dir="./checkpoints/") +model = loader.get_model() +tokenizer = loader.get_tokenizer() +with torch.cuda.device(0): + model = bminf.wrapper(model, quantization=False, memory_limit=30 << 39) + +tokenizer = Tokenizer.from_pretrained(model_name) +predictor = Predictor(model, tokenizer) + +text = "今天天气不错[gMASK]" +output = predictor.predict_generate_randomsample(text, out_max_length=10) +print(text, '\n', output) \ No newline at end of file diff --git a/examples/bminf_generate/gpt2_generate.py b/examples/bminf_generate/gpt2_generate.py new file mode 100644 index 00000000..f05c7ba9 --- /dev/null +++ b/examples/bminf_generate/gpt2_generate.py @@ -0,0 +1,35 @@ +# Copyright © 2022 BAAI. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License") +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +import bminf +import time + +if __name__ == '__main__': + + loader = AutoLoader("seq2seq", + "GPT2-base-ch", + model_dir="./checkpoints/") + model = loader.get_model() + model = model.to('cpu') + tokenizer = loader.get_tokenizer() + time_start=time.time() + with torch.cuda.device(0): + model = bminf.wrapper(model, quantization=False, memory_limit=20 << 30) + predictor = Predictor(model, tokenizer) + + text = "今天天气不错" + + out_2 = predictor.predict_generate_randomsample(text, + input_max_length=512, + out_max_length=100, + repetition_penalty=1.5, + top_k=20, + top_p=0.8) + + time_end=time.time() + print('time cost',time_end-time_start,'s') + # print(f"out_1 is {out_1}") + print(f"out_2 is {out_2}") diff --git a/examples/bminf_generate/gpt2_generate_original.py b/examples/bminf_generate/gpt2_generate_original.py new file mode 100644 index 00000000..54ada5c6 --- /dev/null +++ b/examples/bminf_generate/gpt2_generate_original.py @@ -0,0 +1,35 @@ +# Copyright © 2022 BAAI. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License") +import torch +from flagai.auto_model.auto_loader import AutoLoader +from flagai.model.predictor.predictor import Predictor +import bminf +import time + + +if __name__ == '__main__': + loader = AutoLoader("seq2seq", + "GPT2-base-ch", + model_dir="./checkpoints/") + model = loader.get_model() + tokenizer = loader.get_tokenizer() + time_start=time.time() + # with torch.cuda.device(0): + # model = bminf.wrapper(model, quantization=False, memory_limit=20 << 30) + model.cuda() + predictor = Predictor(model, tokenizer) + + text = "今天天气不错" + + out_2 = predictor.predict_generate_randomsample(text, + input_max_length=512, + out_max_length=100, + repetition_penalty=1.5, + top_k=20, + top_p=0.8) + + time_end=time.time() + print('time cost',time_end-time_start,'s') + # print(f"out_1 is {out_1}") + print(f"out_2 is {out_2}") diff --git a/examples/cpm3_generation/generation.py b/examples/cpm3_generation/generation.py index 809e7bae..61b3f670 100644 --- a/examples/cpm3_generation/generation.py +++ b/examples/cpm3_generation/generation.py @@ -103,7 +103,7 @@ def calc_banned_ngram_tokens( return banned_tokens -# min_length_constriant +# min_length_constraint def min_length_constraint(logits, cur_len, min_len, tokenizer): # This enforcing a min-length by setting EOS probability to 0. if cur_len <= min_len: diff --git a/examples/cpm3_pretrain/data_analyze.py b/examples/cpm3_pretrain/data_analyze.py index d295eb60..b5fb3a3f 100644 --- a/examples/cpm3_pretrain/data_analyze.py +++ b/examples/cpm3_pretrain/data_analyze.py @@ -1,6 +1,6 @@ import json -fout = open('{}'.format('/sharefs/baai-mrnd/xw/cpm3_trian_data/cpm3_train_data.jsonl'), "w", encoding='utf-8') +fout = open('{}'.format('/sharefs/baai-mrnd/xw/cpm3_train_data/cpm3_train_data.jsonl'), "w", encoding='utf-8') fin = open('{}'.format('/sharefs/webbrain-lijijie/data/CEPSUM/test_public.jsonl'), 'r', encoding='utf-8') def random_mask(source: str): diff --git a/examples/cpm_1/generate.py b/examples/cpm_1/generate.py index 9482c006..186864f9 100644 --- a/examples/cpm_1/generate.py +++ b/examples/cpm_1/generate.py @@ -9,7 +9,7 @@ loader = AutoLoader(task_name="lm", model_name="CPM-large-ch", - model_dir="./state_dict/") + model_dir="./checkpoints") model = loader.get_model() tokenizer = loader.get_tokenizer() diff --git a/examples/galactica/generate_galactica_1.3b.py b/examples/galactica/generate_galactica_1.3b.py index dd7e67da..ae15cb19 100644 --- a/examples/galactica/generate_galactica_1.3b.py +++ b/examples/galactica/generate_galactica_1.3b.py @@ -22,6 +22,4 @@ repetition_penalty=1.2, temperature=0.7 ) -print(out) - - +print(out) \ No newline at end of file diff --git a/examples/glm_blank_filling/README.md b/examples/glm_blank_filling/README.md index ac1803f0..27a65aa3 100644 --- a/examples/glm_blank_filling/README.md +++ b/examples/glm_blank_filling/README.md @@ -43,17 +43,14 @@ filling task ```python import torch from flagai.model.glm_model import GLMModel -from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.data.tokenizer import Tokenizer from flagai.model.predictor.predictor import Predictor if __name__ == "__main__": """Main training program.""" print('Generate Samples') - tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', - add_block_symbols=True, - add_task_mask=True, - add_decoder_mask=False, - fix_command_token=False) - model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + tokenizer = Tokenizer.from_pretrained(model_name) + model = GLMModel.from_pretrain(model_name=model_name, + download_path="./checkpoints") model.cuda(torch.cuda.current_device()) predictor = Predictor(model, tokenizer) # question-answering @@ -67,17 +64,14 @@ Similar to BERT, GLM can predict masked tokens as ```python import torch from flagai.model.glm_model import GLMModel -from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.data.tokenizer import Tokenizer from flagai.model.predictor.predictor import Predictor if __name__ == "__main__": """Main training program.""" print('Generate Samples') - tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', - add_block_symbols=True, - add_task_mask=True, - add_decoder_mask=False, - fix_command_token=False) - model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + tokenizer = Tokenizer.from_pretrained(model_name) + model = GLMModel.from_pretrain(model_name=model_name, + download_path="./checkpoints") model.cuda(torch.cuda.current_device()) predictor = Predictor(model, tokenizer) # question-answering @@ -90,17 +84,14 @@ and predict masked sentences as ```python import torch from flagai.model.glm_model import GLMModel -from flagai.data.tokenizer import GLMLargeChTokenizer +from flagai.data.tokenizer import Tokenizer from flagai.model.predictor.predictor import Predictor if __name__ == "__main__": """Main training program.""" print('Generate Samples') - tokenizer = GLMLargeChTokenizer(vocab_path='./checkpoints/glm-large-ch/cog-pretrain.model', - add_block_symbols=True, - add_task_mask=True, - add_decoder_mask=False, - fix_command_token=False) - model = GLMModel.from_pretrain(model_name='glm-large-ch', only_download_config=False) + tokenizer = Tokenizer.from_pretrained(model_name) + model = GLMModel.from_pretrain(model_name=model_name, + download_path="./checkpoints") model.cuda(torch.cuda.current_device()) predictor = Predictor(model, tokenizer) # question-answering diff --git a/examples/glm_blank_filling/glm_generate_samples.py b/examples/glm_blank_filling/glm_generate_samples.py index 40e385a9..01b1bf00 100644 --- a/examples/glm_blank_filling/glm_generate_samples.py +++ b/examples/glm_blank_filling/glm_generate_samples.py @@ -1,7 +1,6 @@ # Copyright © 2022 BAAI. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License") - import torch from flagai.model.glm_model import GLMModel from flagai.data.tokenizer import Tokenizer @@ -43,4 +42,4 @@ for t in text: output = predictor.predict_generate_randomsample( t, top_k=50, repetition_penalty=4.0, top_p=1.0) - print(t, '\n', output) + print(t, '\n', output) \ No newline at end of file diff --git a/examples/glm_blank_filling/glm_generate_samples_en.py b/examples/glm_blank_filling/glm_generate_samples_en.py index 009b4ed1..9fae6140 100644 --- a/examples/glm_blank_filling/glm_generate_samples_en.py +++ b/examples/glm_blank_filling/glm_generate_samples_en.py @@ -1,7 +1,6 @@ # Copyright © 2022 BAAI. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License") - import torch from flagai.model.glm_model import GLMModel from flagai.data.tokenizer import Tokenizer @@ -13,7 +12,7 @@ print('Generate Samples') loader = AutoLoader(task_name='lm', - model_name='GLM-large-en-generation', + model_name='GLM-large-en', only_download_config=False) model = loader.get_model() tokenizer = loader.get_tokenizer() diff --git a/examples/gpt2_title_generation/train_multi_gpu.py b/examples/gpt2_title_generation/train_multi_gpu.py index 7ad121db..a0ed862d 100644 --- a/examples/gpt2_title_generation/train_multi_gpu.py +++ b/examples/gpt2_title_generation/train_multi_gpu.py @@ -1,7 +1,6 @@ # Copyright © 2022 BAAI. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License") -import sys import os import torch from torch.utils.data import Dataset @@ -39,7 +38,7 @@ cur_dir = os.path.dirname(os.path.abspath(__file__)) src_dir = cur_dir + '/data/train.src' tgt_dir = cur_dir + '/data/train.tgt' -model_dir = "./state_dict/" +model_dir = "./checkpoints/" os.makedirs(model_dir, exist_ok=True) maxlen = 256 diff --git a/examples/roberta_ner/generate.py b/examples/roberta_ner/generate.py index 4a7e4850..79b2dbb1 100644 --- a/examples/roberta_ner/generate.py +++ b/examples/roberta_ner/generate.py @@ -7,11 +7,13 @@ device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + task_name = "ner" -model_dir = "./state_dict/" +model_dir = "./checkpoints" # Note "./checkpoints_ner/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_ner. -model_save_path = "./checkpoints_ner/3913/mp_rank_00_model_states.pt" +# model_save_path = "./checkpoints_ner/3913/mp_rank_00_model_states.pt" target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"] @@ -25,8 +27,8 @@ tokenizer = auto_loader.get_tokenizer() predictor = Predictor(model, tokenizer) -model.load_state_dict( - torch.load(model_save_path, map_location=device)["module"]) +# model.load_state_dict( +# torch.load(model_save_path, map_location=device)["module"]) model.to(device) model.eval() diff --git a/examples/roberta_ner/train.py b/examples/roberta_ner/train.py index 74d279ad..d795793b 100644 --- a/examples/roberta_ner/train.py +++ b/examples/roberta_ner/train.py @@ -69,7 +69,7 @@ def load_data(filename): val_data = load_data(valid_path) test_data = load_data(test_path) -print(f"trian_data is {len(train_data)}") +print(f"train_data is {len(train_data)}") print(f"val_data is {len(val_data)}") print(f"test_data is {len(test_data)}") print(f"target is {target}") diff --git a/examples/roberta_ner/train_crf.py b/examples/roberta_ner/train_crf.py index 882745c0..36bfaa09 100644 --- a/examples/roberta_ner/train_crf.py +++ b/examples/roberta_ner/train_crf.py @@ -65,7 +65,7 @@ def load_data(filename): val_data = load_data(valid_path) test_data = load_data(test_path) -print(f"trian_data is {len(train_data)}") +print(f"train_data is {len(train_data)}") print(f"val_data is {len(val_data)}") print(f"test_data is {len(test_data)}") print(f"target is {target}") diff --git a/examples/roberta_ner/train_global_pointer.py b/examples/roberta_ner/train_global_pointer.py index 26bfcd02..43fc627b 100644 --- a/examples/roberta_ner/train_global_pointer.py +++ b/examples/roberta_ner/train_global_pointer.py @@ -61,7 +61,7 @@ def load_data(filename): val_data = load_data(valid_path) test_data = load_data(test_path) -print(f"trian_data is {len(train_data)}") +print(f"train_data is {len(train_data)}") print(f"val_data is {len(val_data)}") print(f"test_data is {len(test_data)}") print(f"target is {target}") diff --git a/examples/roberta_semantic_matching/train.py b/examples/roberta_semantic_matching/train.py index e0648063..30e9821f 100644 --- a/examples/roberta_semantic_matching/train.py +++ b/examples/roberta_semantic_matching/train.py @@ -27,7 +27,7 @@ cur_dir = os.path.dirname(os.path.abspath(__file__)) train_path = cur_dir + "/data/train.tsv" -model_dir = "./state_dict/" +model_dir = "./checkpoints/" maxlen = 256 auto_loader = AutoLoader("semantic-matching", diff --git a/examples/roberta_title_generation/generate.py b/examples/roberta_title_generation/generate.py index c28960be..00dbdf8e 100644 --- a/examples/roberta_title_generation/generate.py +++ b/examples/roberta_title_generation/generate.py @@ -7,7 +7,7 @@ device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -model_dir = "./state_dict" +model_dir = "./checkpoints" # Note "./checkpoints_seq2seq/{}/mp_rank_00_model_states.pt", {} is a directory in the checkpoints_seq2seq. model_save_path = "./checkpoints_seq2seq/10/mp_rank_00_model_states.pt" diff --git a/examples/t5_title_generation/generate.py b/examples/t5_title_generation/generate.py index f6e58391..53bbceca 100644 --- a/examples/t5_title_generation/generate.py +++ b/examples/t5_title_generation/generate.py @@ -5,7 +5,7 @@ from flagai.model.predictor.predictor import Predictor if __name__ == '__main__': - loader = AutoLoader("title-generation", "T5-base-ch", model_dir="./state_dict/") + loader = AutoLoader("title-generation", "T5-base-ch", model_dir="./checkpoints") model = loader.get_model() tokenizer = loader.get_tokenizer() predictor = Predictor(model, tokenizer) diff --git a/flagai/auto_model/auto_loader.py b/flagai/auto_model/auto_loader.py index 7fea3564..cfbbe0db 100644 --- a/flagai/auto_model/auto_loader.py +++ b/flagai/auto_model/auto_loader.py @@ -46,6 +46,7 @@ def __getattr__(self, name): "cpm_lm": ("flagai.model.gpt2_model", "GPT2Model"), "t5_seq2seq": ["flagai.model.t5_model", "T5Model"], "t5_lm": ["flagai.model.t5_model", "T5Model"], + "t5_title-generation": ["flagai.model.t5_model", "T5Model"], "alm_lm": ["flagai.model.alm_model", "ALMModel"], "glm_lm": ["flagai.model.glm_model", "GLMModel"], "glm_seq2seq": ["flagai.model.glm_model", "GLMForSeq2Seq"], @@ -62,7 +63,7 @@ def __getattr__(self, name): "swinv2_classification": ("flagai.model.vision.swinv2", "SwinTransformerV2"), "cpm3_lm": ("flagai.model.cpm3_model", "CPM3"), - "cpm3_trian": ("flagai.model.cpm3_trian_model", "CPM3"), + "cpm3_train": ("flagai.model.cpm3_train_model", "CPM3"), "diffusion_text2img": ("flagai.model.mm.AltDiffusion", "LatentDiffusion"), "altclip_txt_img_matching": ("flagai.model.mm.AltCLIP", "AltCLIP"), "evaclip_txt_img_matching": ("flagai.model.mm.eva_clip_model", "EVA_CLIP"), diff --git a/flagai/data/dataset/block/blocklm_utils.py b/flagai/data/dataset/block/blocklm_utils.py index 4687305f..44fda3d2 100644 --- a/flagai/data/dataset/block/blocklm_utils.py +++ b/flagai/data/dataset/block/blocklm_utils.py @@ -86,10 +86,10 @@ def __init__(self, self.encoder_decoder = encoder_decoder self.shuffle_blocks = shuffle_blocks self.sentinel_token = sentinel_token - self.generation_mask = 'gMASK' if task_mask else 'MASK' + self.generation_mask = 'gMASK' if task_mask else 'mask' self.generation_mask = self.tokenizer.get_command_id( self.generation_mask) - self.gap_sentence_mask = 'sMASK' if task_mask else 'MASK' + self.gap_sentence_mask = 'sMASK' if task_mask else 'mask' self.gap_sentence_mask = self.tokenizer.get_command_id( self.gap_sentence_mask) self.random_position = random_position @@ -205,7 +205,7 @@ def make_masked_data(self, # position_ids = np.arange(len(tokens), dtype=np.int64) targets = copy.deepcopy(tokens) - mask_id = self.tokenizer.get_command_id('MASK') + mask_id = self.tokenizer.get_command_id('mask') mlm_masks = np.zeros(len(tokens), dtype=np.int64) for start, end in block_spans: for idx in range(start, end): @@ -273,7 +273,7 @@ def make_block_data(self, elif task == 'gap_sentence': mask_id = self.gap_sentence_mask else: - mask_token = 'MASK' if idx == 0 else f'MASK{idx}' + mask_token = 'mask' if idx == 0 else f'MASK{idx}' mask_id = self.tokenizer.get_command_id(mask_token) local_spans.append((current_length, current_length + start - last)) source_tokens.append(tokens[last:start]) diff --git a/flagai/data/dataset/data_collator/collate_fn.py b/flagai/data/dataset/data_collator/collate_fn.py index 73b2f8e5..6eb629d5 100644 --- a/flagai/data/dataset/data_collator/collate_fn.py +++ b/flagai/data/dataset/data_collator/collate_fn.py @@ -126,7 +126,7 @@ def __init__(self, args, tokenizer, task_name): def encode(self, example): cls_id = self.tokenizer.get_command_id('cls') - mask_token = 'sMASK' if self.args.task_mask else 'MASK' + mask_token = 'sMASK' if self.args.task_mask else 'mask' mask_id = self.tokenizer.get_command_id(mask_token) pad_id = self.tokenizer.get_command_id('pad') sop_id = self.tokenizer.get_command_id('sop') @@ -175,7 +175,7 @@ def sub_finder(mylist, pattern): source_tokens = [cls_id] + source_tokens + [mask_id ] + answer_tokens elif self.task_name in ["cmrc"]: - mask_id = self.tokenizer.get_command_id('MASK') + mask_id = self.tokenizer.get_command_id('mask') source_text = example.text_a target_text = example.meta["answer"].strip() question = example.meta["question"].strip() @@ -191,7 +191,7 @@ def sub_finder(mylist, pattern): mask_id ] + source_tokens[:max_src_length] elif self.task_name in ["wsc"]: - mask_id = self.tokenizer.get_command_id('MASK') + mask_id = self.tokenizer.get_command_id('mask') source_text = example.text_a target_text = example.meta["answer"].strip() question = example.meta["question"].strip() @@ -307,10 +307,10 @@ def __init__(self, self.encoder_decoder = encoder_decoder self.shuffle_blocks = shuffle_blocks self.sentinel_token = sentinel_token - self.generation_mask = 'gMASK' if task_mask else 'MASK' + self.generation_mask = 'gMASK' if task_mask else 'mask' self.generation_mask = self.tokenizer.get_command_id( self.generation_mask) - self.gap_sentence_mask = 'sMASK' if task_mask else 'MASK' + self.gap_sentence_mask = 'sMASK' if task_mask else 'mask' self.gap_sentence_mask = self.tokenizer.get_command_id( self.gap_sentence_mask) self.random_position = random_position @@ -426,7 +426,7 @@ def make_masked_data(self, position_ids = np.arange(len(tokens), dtype=np.int64) targets = copy.deepcopy(tokens) - mask_id = self.tokenizer.get_command_id('MASK') + mask_id = self.tokenizer.get_command_id('mask') mlm_masks = np.zeros(len(tokens), dtype=np.int64) for start, end in block_spans: for idx in range(start, end): @@ -494,7 +494,7 @@ def make_block_data(self, elif task == 'gap_sentence': mask_id = self.gap_sentence_mask else: - mask_token = 'MASK' if idx == 0 else f'MASK{idx}' + mask_token = 'mask' if idx == 0 else f'MASK{idx}' mask_id = self.tokenizer.get_command_id(mask_token) local_spans.append((current_length, current_length + start - last)) source_tokens.append(tokens[last:start]) diff --git a/flagai/data/dataset/data_utils.py b/flagai/data/dataset/data_utils.py index 4f0ee38d..1efee372 100644 --- a/flagai/data/dataset/data_utils.py +++ b/flagai/data/dataset/data_utils.py @@ -134,7 +134,7 @@ def build_input_from_ids(text_a_ids, # Prepare ids for special tokens if mask_id is None: - mask_id = tokenizer.get_command_id('MASK') + mask_id = tokenizer.get_command_id('mask') eos_id = tokenizer.get_command_id('eos') # end of sentence token cls_id = tokenizer.get_command_id('cls') # start of sentence token sep_id = tokenizer.get_command_id('sep') # seperator of two texts token @@ -235,7 +235,7 @@ def build_input_from_ids(text_a_ids, # def build_decoder_input(enc_ids, answer_ids, max_seq_length, max_dec_seq_length, tokenizer): - mask_id = tokenizer.get_command_id('MASK') + mask_id = tokenizer.get_command_id('mask') eos_id = tokenizer.get_command_id('eos') sop_id = tokenizer.get_command_id('sop') masks = [] diff --git a/flagai/data/dataset/language_model/dataset.py b/flagai/data/dataset/language_model/dataset.py index b291251b..a911df81 100644 --- a/flagai/data/dataset/language_model/dataset.py +++ b/flagai/data/dataset/language_model/dataset.py @@ -38,7 +38,7 @@ def __init__(self, args, documents, tokenizer, num_original_tokens, self.left_weights = [0] + self.weights[:-1] self.unidirectional = args.unidirectional self.block_lm = args.block_lm - mask_token = "gMASK" if args.task_mask else 'MASK' + mask_token = "gMASK" if args.task_mask else 'mask' self.mask_id = self.tokenizer.get_command_id(mask_token) def __len__(self): @@ -115,7 +115,7 @@ def __init__(self, args, tokenizer, strict=True): self.strict = strict self.block_lm = args.block_lm self.unidirectional = args.unidirectional - mask_token = "gMASK" if args.task_mask else 'MASK' + mask_token = "gMASK" if args.task_mask else 'mask' self.mask_id = self.tokenizer.get_command_id(mask_token) self.tokens = [] diff --git a/flagai/data/dataset/seq2seq/dataset.py b/flagai/data/dataset/seq2seq/dataset.py index adc28149..b0bc4148 100644 --- a/flagai/data/dataset/seq2seq/dataset.py +++ b/flagai/data/dataset/seq2seq/dataset.py @@ -477,7 +477,7 @@ def __len__(self): def __getitem__(self, idx): example = self.example_list[idx] source_text, target_text = example.text_a, example.text_b - mask_token = 'MASK' + mask_token = 'mask' mask_id = self.tokenizer.get_command_id(mask_token) sop_id = self.tokenizer.get_command_id('sop') eop_id = self.tokenizer.get_command_id('eop') @@ -612,7 +612,7 @@ def __len__(self): def __getitem__(self, idx): example = self.example_list[idx] source_text = example.text_a - mask_token = 'gMASK' if self.args.task_mask else 'MASK' + mask_token = 'gMASK' if self.args.task_mask else 'mask' mask_id = self.tokenizer.get_command_id(mask_token) sop_id = self.tokenizer.get_command_id('sop') eop_id = self.tokenizer.get_command_id('eop') diff --git a/flagai/data/dataset/superglue/pvp.py b/flagai/data/dataset/superglue/pvp.py index d4d07b39..8a4d6ee3 100644 --- a/flagai/data/dataset/superglue/pvp.py +++ b/flagai/data/dataset/superglue/pvp.py @@ -97,12 +97,12 @@ def spell_length(self): @property def mask(self) -> str: """Return the underlying LM's mask token""" - return self.tokenizer.get_command_id('MASK') + return self.tokenizer.get_command_id('mask') @property def mask_id(self) -> int: """Return the underlying LM's mask id""" - return self.tokenizer.get_command_id('MASK') + return self.tokenizer.get_command_id('mask') @property def max_num_verbalizers(self) -> int: @@ -574,13 +574,13 @@ def spell_length(self): @property def mask(self) -> str: """Return the underlying LM's mask token""" - mask_token = 'MASK' + mask_token = 'mask' return self.tokenizer.get_command_id(mask_token) @property def mask_id(self) -> int: """Return the underlying LM's mask id""" - mask_token = 'MASK' + mask_token = 'mask' return self.tokenizer.get_command_id(mask_token) def get_answers(self, example: InputExample): diff --git a/flagai/data/file_utils.py b/flagai/data/file_utils.py index 15ebf0e9..40af5d47 100644 --- a/flagai/data/file_utils.py +++ b/flagai/data/file_utils.py @@ -20,7 +20,6 @@ from hashlib import sha256 import sys from io import open - import boto3 import requests from botocore.exceptions import ClientError diff --git a/flagai/data/tokenizer/__init__.py b/flagai/data/tokenizer/__init__.py index e07653af..c780a6b8 100644 --- a/flagai/data/tokenizer/__init__.py +++ b/flagai/data/tokenizer/__init__.py @@ -6,5 +6,6 @@ from .bert.bert_tokenizer import BertWordPieceTokenizer from .cpm_1.cpm1_tokenizer import CPMTokenizer from .opt.opt_en_tokenizer import OPTTokenizer +from .t5.t5_pegasus_tokenizer import T5PegasusTokenizer from .uni_tokenizer.tokenizer import Tokenizer # from .uni_tokenizer.base_tokenizer import BaseTokenizer diff --git a/flagai/data/tokenizer/bert/bert_tokenizer.py b/flagai/data/tokenizer/bert/bert_tokenizer.py index 0ba3fdf6..3c935713 100644 --- a/flagai/data/tokenizer/bert/bert_tokenizer.py +++ b/flagai/data/tokenizer/bert/bert_tokenizer.py @@ -74,8 +74,8 @@ def __init__(self, tokenizer_model_type=None, cache_dir=None): self._command_tokens = [ CommandToken('pad', '[PAD]', self.get_specialid_from_text_tokenizer('pad')), - CommandToken('ENC', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), - CommandToken('MASK', '[MASK]', + CommandToken('cls', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), + CommandToken('mask', '[MASK]', self.get_specialid_from_text_tokenizer('mask')), CommandToken('unk', '[UNK]', self.get_specialid_from_text_tokenizer('unk')), CommandToken('sep', '[SEP]', self.get_specialid_from_text_tokenizer('sep')), diff --git a/flagai/data/tokenizer/cpm_1/cpm1_tokenizer.py b/flagai/data/tokenizer/cpm_1/cpm1_tokenizer.py index 7bae0deb..24275e1f 100644 --- a/flagai/data/tokenizer/cpm_1/cpm1_tokenizer.py +++ b/flagai/data/tokenizer/cpm_1/cpm1_tokenizer.py @@ -37,7 +37,7 @@ def __init__(self, vocab_file, model_file, max_length=None): self.encoder = json.load(open(vocab_file)) self.decoder = {v: k for k, v in self.encoder.items()} - self.sp = spm.SentencePieceProcessor(model_file=model_file) + self.sp_model = spm.SentencePieceProcessor(model_file=model_file) self.translator = str.maketrans(" \n", "\u2582\u2583") self.token_start_id = 0 self.token_end_id = 3 @@ -48,6 +48,13 @@ def __init__(self, vocab_file, model_file, max_length=None): def vocab_size(self): return len(self.encoder) + def get_vocab(self): + vocab = { + self.convert_id_to_token(i): i + for i in range(self.vocab_size) + } + return vocab + def __len__(self): return len(self.encoder) + len(self.special_tokens) @@ -57,19 +64,28 @@ def eod(self): def tokenize(self, text): """ Tokenize a string. """ - seg_list = [ - x.translate(self.translator) - for x in jieba.cut(text, cut_all=False) - ] - new_seg = " ".join(seg_list) - return self.sp.encode(new_seg) + seg_list = [x.translate(self.translator) for x in jieba.cut(text, cut_all=False)] + new_seg = "".join(seg_list) + return self.sp_model.encode(new_seg) def encode(self, text): res = self.tokenize(text) return res + + def convert_tokens_to_ids(self, tokens): + return [self.sp_model.PieceToId(token) for token in tokens] + + def convert_token_to_id(self, token): + return self.sp_model.PieceToId(token) + + def convert_id_to_token(self, idx): + return self.sp_model.IdToPiece(int(idx)) + + def convert_ids_to_tokens(self, idxs): + return [self.sp_model.IdToPiece(int(idx)) for idx in idxs] def decode(self, tokens): - text = self.sp.decode(tokens) + text = self.sp_model.decode(tokens) text = text.replace(' ', '').replace('\u2582', ' ').replace('\u2583', '\n') return text @@ -78,3 +94,18 @@ def encode_plus(self, text, max_length=None): res = self.encode(text) return {"input_ids": res} + + def convert_tokens_to_string(self, tokens, all_command_token={}): + """Converts a sequence of tokens (string) in a single string.""" + current_sub_tokens = [] + out_string = "" + for token in tokens: + # make sure that special tokens are not decoded using sentencepiece model + if token in all_command_token: + out_string += self.sp_model.decode_pieces( + current_sub_tokens) + token + " " + current_sub_tokens = [] + else: + current_sub_tokens.append(token) + out_string += self.sp_model.decode_pieces(current_sub_tokens) + return out_string.strip() diff --git a/flagai/data/tokenizer/galactica/__init__.py b/flagai/data/tokenizer/galactica/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/flagai/data/tokenizer/galactica/__init__.py @@ -0,0 +1 @@ + diff --git a/flagai/data/tokenizer/galactica/galactica_tokenizer.py b/flagai/data/tokenizer/galactica/galactica_tokenizer.py index 87a28412..f028d0f0 100644 --- a/flagai/data/tokenizer/galactica/galactica_tokenizer.py +++ b/flagai/data/tokenizer/galactica/galactica_tokenizer.py @@ -14,8 +14,8 @@ def __init__(self, download_dir) -> None: self._command_tokens = [ CommandToken('pad', '[PAD]', self.get_specialid_from_text_tokenizer('pad')), - CommandToken('ENC', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), - CommandToken('MASK', '[MASK]', + CommandToken('cls', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), + CommandToken('mask', '[MASK]', self.get_specialid_from_text_tokenizer('mask')), CommandToken('unk', '[UNK]', self.get_specialid_from_text_tokenizer('unk')), CommandToken('sep', '[SEP]', self.get_specialid_from_text_tokenizer('sep')), diff --git a/flagai/data/tokenizer/glm_10b_en/glm_10b_en_bpe_tokenizer.py b/flagai/data/tokenizer/glm_10b_en/glm_10b_en_bpe_tokenizer.py index b762b66b..e592d33d 100644 --- a/flagai/data/tokenizer/glm_10b_en/glm_10b_en_bpe_tokenizer.py +++ b/flagai/data/tokenizer/glm_10b_en/glm_10b_en_bpe_tokenizer.py @@ -60,7 +60,7 @@ def __init__(self, self.text_tokenizer.encoder['']), CommandToken('cls', '[CLS]', self.text_tokenizer.encoder['']), - CommandToken('MASK', + CommandToken('mask', '[MASK]', self.text_tokenizer.encoder[''], lstrip=True), @@ -88,7 +88,7 @@ def __init__(self, CommandToken('sop', '<|startofpiece|>', self.num_tokens), CommandToken('eop', '<|endofpiece|>', self.num_tokens + 1), CommandToken('cls', '[CLS]', self.num_tokens + 2), - CommandToken('MASK', + CommandToken('mask', '[MASK]', self.num_tokens + 3, lstrip=True), diff --git a/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py b/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py index 69048d3a..b91797f6 100644 --- a/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py +++ b/flagai/data/tokenizer/glm_large_ch/glm_large_ch_tokenizer.py @@ -55,7 +55,7 @@ def __init__(self, CommandToken('eos', '<|endoftext|>', self.num_text_tokens), CommandToken('sep', '[SEP]', self.num_text_tokens + 1), CommandToken('cls', '[CLS]', self.num_text_tokens + 2), - CommandToken('MASK', + CommandToken('mask', '[MASK]', self.num_text_tokens + 3, lstrip=True), diff --git a/flagai/data/tokenizer/glm_large_en/glm_large_en_tokenizer.py b/flagai/data/tokenizer/glm_large_en/glm_large_en_tokenizer.py index ff4e1e4a..db4c726f 100644 --- a/flagai/data/tokenizer/glm_large_en/glm_large_en_tokenizer.py +++ b/flagai/data/tokenizer/glm_large_en/glm_large_en_tokenizer.py @@ -59,7 +59,7 @@ def __init__(self, self._command_tokens = [ CommandToken('pad', '[PAD]', self.text_tokenizer.vocab['[PAD]']), CommandToken('cls', '[CLS]', self.text_tokenizer.vocab['[CLS]']), - CommandToken('MASK', '[MASK]', + CommandToken('mask', '[MASK]', self.text_tokenizer.vocab['[MASK]']), CommandToken('unk', '[UNK]', self.text_tokenizer.vocab['[UNK]']), CommandToken('sep', '[SEP]', self.text_tokenizer.vocab['[SEP]']), diff --git a/flagai/data/tokenizer/opt/opt_en_tokenizer.py b/flagai/data/tokenizer/opt/opt_en_tokenizer.py index 8501601a..9e8e528c 100644 --- a/flagai/data/tokenizer/opt/opt_en_tokenizer.py +++ b/flagai/data/tokenizer/opt/opt_en_tokenizer.py @@ -34,8 +34,8 @@ def __init__(self, tokenizer_model_type="facebook/opt-125m", cache_dir=None): self._command_tokens = [ CommandToken('pad', '[PAD]', self.get_specialid_from_text_tokenizer('pad')), - CommandToken('ENC', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), - CommandToken('MASK', '[MASK]', + CommandToken('cls', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), + CommandToken('mask', '[MASK]', self.get_specialid_from_text_tokenizer('mask')), CommandToken('unk', '[UNK]', self.get_specialid_from_text_tokenizer('unk')), CommandToken('sep', '[SEP]', self.get_specialid_from_text_tokenizer('sep')), diff --git a/flagai/data/tokenizer/roberta/roberta_tokenizer.py b/flagai/data/tokenizer/roberta/roberta_tokenizer.py index a525f2a6..f1b270e4 100644 --- a/flagai/data/tokenizer/roberta/roberta_tokenizer.py +++ b/flagai/data/tokenizer/roberta/roberta_tokenizer.py @@ -37,8 +37,8 @@ def __init__(self, tokenizer_model_type="roberta-base", cache_dir=None): self._command_tokens = [ CommandToken('pad', '[PAD]', self.get_specialid_from_text_tokenizer('pad')), - CommandToken('ENC', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), - CommandToken('MASK', '[MASK]', + CommandToken('cls', '[CLS]', self.get_specialid_from_text_tokenizer('cls')), + CommandToken('mask', '[MASK]', self.get_specialid_from_text_tokenizer('mask')), CommandToken('unk', '[UNK]', self.get_specialid_from_text_tokenizer('unk')), CommandToken('sep', '[SEP]', self.get_specialid_from_text_tokenizer('sep')), diff --git a/flagai/data/tokenizer/t5/t5_tokenizer.py b/flagai/data/tokenizer/t5/t5_tokenizer.py index 8774b3af..499aa83e 100644 --- a/flagai/data/tokenizer/t5/t5_tokenizer.py +++ b/flagai/data/tokenizer/t5/t5_tokenizer.py @@ -44,8 +44,8 @@ def __init__(self, tokenizer_model_type="t5-base", cache_dir=None): CommandToken('sep', '[SEP]', self.num_tokens), CommandToken('pad', '[PAD]', self.num_tokens + 1), - CommandToken('ENC', '[CLS]', self.num_tokens + 2), - CommandToken('MASK', '[MASK]', + CommandToken('cls', '[CLS]', self.num_tokens + 2), + CommandToken('mask', '[MASK]', self.num_tokens + 3), ] self._command_tokens.extend([ diff --git a/flagai/data/tokenizer/tokenizer.py b/flagai/data/tokenizer/tokenizer.py index 3f82e7f5..43585688 100644 --- a/flagai/data/tokenizer/tokenizer.py +++ b/flagai/data/tokenizer/tokenizer.py @@ -54,7 +54,7 @@ def __str__(self): ('sep', 4), ('L2R', 5), ('cls', 6), - ('MASK', 7), + ('mask', 7), ] DEFAULT_COMMAND_TOKENS = prep_command_tokens(DEFAULT_COMMAND_TOKENS) """define some default type tokens for bert training""" @@ -457,8 +457,12 @@ def DecodeTokens(self, tokens): """A list of tokens => recovered text string""" return self.text_tokenizer.convert_tokens_to_string(tokens) + def convert_tokens_to_ids(self, tokens): + return self.text_tokenizer.convert_tokens_to_ids(tokens) + + def convert_ids_to_tokens(self, ids): + return self.text_tokenizer.convert_ids_to_tokens(ids) -# class BaseTokenizer(object): class TextTokenizer(object): """ diff --git a/flagai/data/tokenizer/uni_tokenizer/base_tokenizer.py b/flagai/data/tokenizer/uni_tokenizer/base_tokenizer.py index dca6bb82..37629623 100644 --- a/flagai/data/tokenizer/uni_tokenizer/base_tokenizer.py +++ b/flagai/data/tokenizer/uni_tokenizer/base_tokenizer.py @@ -1,6 +1,6 @@ import os from flagai.model.file_utils import _get_model_files, _get_model_id, _get_vocab_path -from flagai.data.tokenizer.uni_tokenizer.properties import VOCAB_FILE, MERGES_FILE, SP_MODEL_FILE, VOCAB_JSON_FILE +from flagai.data.tokenizer.uni_tokenizer.properties import VOCAB_FILE, MERGES_FILE, SP_MODEL_FILE, VOCAB_JSON_FILE, TOKENIZER_JSON_FILE, SPECIAL_TOKENS_MAP import warnings @@ -63,10 +63,13 @@ def from_pretrained(cls, resolved_vocab_file = os.path.join(cache_dir, VOCAB_FILE) resolved_merges_file = os.path.join(cache_dir, MERGES_FILE) resolved_sp_file = os.path.join(cache_dir, SP_MODEL_FILE) + special_tokens_map = os.path.join(cache_dir, SPECIAL_TOKENS_MAP) + resolved_tokenizer_json_file = os.path.join(cache_dir, TOKENIZER_JSON_FILE) if tokenizer_class == "wp": return cls(vocab_file=resolved_vocab_file, tokenizer_class=tokenizer_class, tokenizer_model_name=tokenizer_model_name, + special_tokens_map=special_tokens_map, cache_dir=cache_dir, *inputs, **kwargs) @@ -75,13 +78,17 @@ def from_pretrained(cls, merges_file=resolved_merges_file, tokenizer_class=tokenizer_class, tokenizer_model_name=tokenizer_model_name, + special_tokens_map=special_tokens_map, cache_dir=cache_dir, *inputs, **kwargs) elif tokenizer_class == "sp": - return cls(sp_model_file=resolved_sp_file, + return cls(vocab_file=resolved_vocab_json_file, + sp_model_file=resolved_sp_file, tokenizer_class=tokenizer_class, tokenizer_model_name=tokenizer_model_name, + tokenizer_json_file=resolved_tokenizer_json_file, + special_tokens_map=special_tokens_map, cache_dir=cache_dir, *inputs, **kwargs) @@ -94,8 +101,10 @@ def __init__(self, vocab_file=None, merges_file=None, sp_model_file=None, + tokenizer_json_file=None, tokenizer_class=None, tokenizer_model_name=None, + special_tokens_map=None, cache_dir=None, *inputs, **kwargs): @@ -105,5 +114,7 @@ def __init__(self, self.sp_model_file = sp_model_file self.tokenizer_class = tokenizer_class self.tokenizer_model_name = tokenizer_model_name + self.tokenizer_json_file = tokenizer_json_file + self.special_tokens_map = special_tokens_map self.cache_dir = cache_dir self.deprecation_warnings = ({}) diff --git a/flagai/data/tokenizer/uni_tokenizer/bpe_tokenizer.py b/flagai/data/tokenizer/uni_tokenizer/bpe_tokenizer.py index 25883388..1f175e16 100644 --- a/flagai/data/tokenizer/uni_tokenizer/bpe_tokenizer.py +++ b/flagai/data/tokenizer/uni_tokenizer/bpe_tokenizer.py @@ -42,6 +42,7 @@ def lru_cache(): return lambda func: func + class BPETokenizer(object): def __init__(self, @@ -151,7 +152,7 @@ def tokenize(self, text): def convert_token_to_id(self, token): """ Converts a sequence of tokens into ids using the vocab. """ - return self.encoder.get(token, 0) + return self.encoder[token] def convert_tokens_to_ids(self, tokens): """ Converts a sequence of tokens into ids using the vocab. """ diff --git a/flagai/data/tokenizer/uni_tokenizer/difffusion_bert_tokenizer.py b/flagai/data/tokenizer/uni_tokenizer/diffusion_bert_tokenizer.py similarity index 100% rename from flagai/data/tokenizer/uni_tokenizer/difffusion_bert_tokenizer.py rename to flagai/data/tokenizer/uni_tokenizer/diffusion_bert_tokenizer.py diff --git a/flagai/data/tokenizer/uni_tokenizer/properties.py b/flagai/data/tokenizer/uni_tokenizer/properties.py index 78499629..aa841bfa 100644 --- a/flagai/data/tokenizer/uni_tokenizer/properties.py +++ b/flagai/data/tokenizer/uni_tokenizer/properties.py @@ -2,4 +2,6 @@ VOCAB_JSON_FILE = 'vocab.json' MERGES_FILE = 'merges.txt' SP_MODEL_FILE = 'spiece.model' -SPECIAL_TOKENS_NAME = 'special_tokens.txt' \ No newline at end of file +TOKENIZER_JSON_FILE = 'tokenizer.json' +SPECIAL_TOKENS_NAME = 'special_tokens.txt' +SPECIAL_TOKENS_MAP = 'special_tokens_map.json' \ No newline at end of file diff --git a/flagai/data/tokenizer/uni_tokenizer/sp_tokenizer.py b/flagai/data/tokenizer/uni_tokenizer/sp_tokenizer.py index 8d817142..e38e7ec5 100644 --- a/flagai/data/tokenizer/uni_tokenizer/sp_tokenizer.py +++ b/flagai/data/tokenizer/uni_tokenizer/sp_tokenizer.py @@ -29,8 +29,6 @@ def __init__(self, model_path): self.sp_model = spm.SentencePieceProcessor() self.sp_model.Load(model_path) # vocab = self.get_vocab() - # print(vocab["<|endoftext|>"]) - # print(vocab["<|endofpiece|>"]) @property def vocab_size(self): diff --git a/flagai/data/tokenizer/uni_tokenizer/tokenizer.py b/flagai/data/tokenizer/uni_tokenizer/tokenizer.py index b27bb24d..4f0c0cde 100644 --- a/flagai/data/tokenizer/uni_tokenizer/tokenizer.py +++ b/flagai/data/tokenizer/uni_tokenizer/tokenizer.py @@ -28,9 +28,10 @@ from flagai.data.tokenizer.uni_tokenizer.bpe_tokenizer import BPETokenizer, MMBPETokenizer from flagai.data.tokenizer.uni_tokenizer.sp_tokenizer import SentencePieceTokenizer from flagai.data.tokenizer.uni_tokenizer.base_tokenizer import BaseTokenizer -from flagai.data.tokenizer.uni_tokenizer.difffusion_bert_tokenizer import FullTokenizer +from flagai.data.tokenizer.uni_tokenizer.diffusion_bert_tokenizer import FullTokenizer from typing import List, Union, Optional import unicodedata +import json def is_control(ch): @@ -38,7 +39,6 @@ def is_control(ch): https://en.wikipedia.org/wiki/Control_character https://www.fileformat.info/info/unicode/category/Cc/index.htm https://www.fileformat.info/info/unicode/category/Cf/index.htm - """ return unicodedata.category(ch) in ('Cc', 'Cf') @@ -50,7 +50,9 @@ def __init__(self, add_sentinel_token=0, add_task_mask=True, add_decoder_mask=False, - fix_command_token=True, + fix_command_token=False, + pre_tokenizer=None, + special_tokens=['cls','pad','unk','eos','sep','mask'], **kwargs): super().__init__(**kwargs) @@ -70,277 +72,88 @@ def __init__(self, self.text_tokenizer = BPETokenizer(self.vocab_file, self.merges_file) elif self.tokenizer_class == "sp": - self.text_tokenizer = SentencePieceTokenizer(self.sp_model_file) + if self.tokenizer_model_name.lower().startswith('cpm'): + from flagai.data.tokenizer.cpm_1.cpm1_tokenizer import CPMTokenizer + self.text_tokenizer = CPMTokenizer(self.vocab_file, self.sp_model_file) + elif self.tokenizer_model_name.lower().startswith('cpm3'): + from flagai.data.tokenizer.cpm_3.cpm3_tokenizer import CPMTokenizer + self.text_tokenizer = CPMTokenizer(self.tokenizer_json_file, self.sp_model_file) + else: + self.text_tokenizer = SentencePieceTokenizer(self.sp_model_file) else: raise NotImplementedError("cannot assign a tokenize class") self.is_glm = self.tokenizer_model_name.lower().startswith('glm') # self.is_clip = self.tokenizer_model_name.startswith('clip') self.num_tokens = self.text_tokenizer.vocab_size - - if self.tokenizer_class == "wp": - # set command tokens from wordpiece tokenizer values - self.num_command_tokens = 6 - self.num_text_tokens = self.num_tokens - 5 - self.num_type_tokens = 2 - self.token_start_id = None - self.token_end_id = None - self.token_pad_id = None - try: - self._command_tokens = [ - CommandToken( - 'pad', '[PAD]', - self.text_tokenizer.convert_token_to_id('[PAD]')), - CommandToken( - 'cls', '[CLS]', - self.text_tokenizer.convert_token_to_id('[CLS]')), - CommandToken( - 'MASK', '[MASK]', - self.text_tokenizer.convert_token_to_id('[MASK]')), - CommandToken( - 'unk', '[UNK]', - self.text_tokenizer.convert_token_to_id('[UNK]')), - CommandToken( - 'sep', '[SEP]', - self.text_tokenizer.convert_token_to_id('[SEP]')), - CommandToken( - 'eos', '[PAD]', - self.text_tokenizer.convert_token_to_id('[PAD]')), - ] - self.token_start_id = self.text_tokenizer.convert_token_to_id( - '[CLS]') - self.token_end_id = self.text_tokenizer.convert_token_to_id( - '[SEP]') - self.token_pad_id = self.text_tokenizer.convert_token_to_id( - '[PAD]') + if self.tokenizer_model_name.startswith('cpm'): + special_tokens.append('eod') + if self.tokenizer_model_name.startswith('opt'): + special_tokens.append('bos') + + try: + with open(self.special_tokens_map, encoding='utf8') as file: dct=json.load(file) + sp_tokens = [(k.replace("_token",""),v['content']) for k,v in dct.items()] + except FileNotFoundError: + dct = None + sp_tokens = [] + for tk in special_tokens: + res = self.search_special(tk) + if res: + sp_tokens += [(tk, res)] + self._command_tokens = [CommandToken(e[0], e[1], self.text_tokenizer.convert_token_to_id(e[1])) for e in sp_tokens] + if self.tokenizer_model_name.lower().startswith("glm"): + if self.tokenizer_class == "wp": self.text_tokenizer._token_cls = "[CLS]" self.text_tokenizer._token_sep = "[SEP]" - - except KeyError: + fix_command_token = False + elif self.tokenizer_class == "sp": + fix_command_token = True self._command_tokens = [ - CommandToken( - 'pad', '[PAD]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'cls', '[CLS]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'MASK', '[MASK]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'unk', '[UNK]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'sep', '[SEP]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'eos', '[PAD]', - self.text_tokenizer.convert_token_to_id('')), + CommandToken('pad', '<|endoftext|>', self.num_tokens), + CommandToken('eos', '<|endoftext|>', self.num_tokens), + CommandToken('sep', '[SEP]', self.num_tokens + 1), + CommandToken('cls', '[CLS]', self.num_tokens + 2), + CommandToken('mask', '[MASK]', self.num_tokens + 3, lstrip=True), + CommandToken('unk', '[UNK]', self.num_tokens + 4) ] - self.token_start_id = self.text_tokenizer.convert_token_to_id( - '') - self.token_end_id = self.text_tokenizer.convert_token_to_id( - '') - self.token_pad_id = self.text_tokenizer.convert_token_to_id( - '') - self.text_tokenizer._token_cls = "" - self.text_tokenizer._token_sep = "" - if add_block_symbols: - self.add_command_token('sop', '<|startofpiece|>') - self.add_command_token('eop', '<|endofpiece|>',) - if add_task_mask: - self.add_command_token('gMASK', '[gMASK]') - self.add_command_token('sMASK', '[sMASK]') - if add_decoder_mask: - self.add_command_token('dBLOCK', '[dBLOCK]') - if add_sentinel_token > 0: - for i in range(1, add_sentinel_token): - self.add_command_token(f'MASK{i}', f'[MASK{i}]') - self.add_command_token(f'sop{i}', f'<|startofpiece{i}|>') - elif self.tokenizer_class == "bpe": - if self.tokenizer_model_name.lower().startswith('roberta'): - self.num_command_tokens = 6 - self.num_text_tokens = self.num_tokens - 3 + self.num_tokens += 6 + elif self.tokenizer_class == "bpe": self._command_tokens = [ - CommandToken( - 'pad', '<|endoftext|>', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'eos', '<|endoftext|>', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'sep', '[SEP]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'cls', '[CLS]', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'MASK', - '[MASK]', - self.text_tokenizer.convert_token_to_id(''), - lstrip=True), - CommandToken( - 'unk', '[UNK]', - self.text_tokenizer.convert_token_to_id('')) + CommandToken('pad', '<|endoftext|>', + self.text_tokenizer.encoder['<|endoftext|>']), + CommandToken('eos', '<|endoftext|>', + self.text_tokenizer.encoder['<|endoftext|>']) ] - if add_block_symbols: - self._command_tokens.extend([ - CommandToken('sop', '<|startofpiece|>', - self.num_tokens), - CommandToken('eop', '<|endofpiece|>', - self.num_tokens + 1) - ]) - self.num_tokens += 2 - self.num_command_tokens += 2 - self.token_end_id = self.text_tokenizer.convert_token_to_id( - '') - elif self.tokenizer_model_name.lower().startswith('clip'): - self.num_command_tokens = 2 - self._command_tokens = [ - CommandToken( - 'sot', '', - self.text_tokenizer.convert_token_to_id('')), - CommandToken( - 'eot', '', - self.text_tokenizer.convert_token_to_id('')), - ] - self.num_tokens += self.num_command_tokens - self.token_end_id = self.text_tokenizer.convert_token_to_id( - '') - else: - self.num_command_tokens = 2 - self.num_text_tokens = self.num_tokens - 1 - self._command_tokens = [ - CommandToken( - 'pad', '<|endoftext|>', - self.text_tokenizer.convert_token_to_id( - '<|endoftext|>')), - CommandToken( - 'eos', '<|endoftext|>', - self.text_tokenizer.convert_token_to_id( - '<|endoftext|>')) - ] - self.token_end_id = self.text_tokenizer.convert_token_to_id( - '<|endoftext|>') - if add_block_symbols: - if self.tokenizer_model_name.lower().startswith('glm'): - unk_token_id = self.num_tokens + 5 - cls_token_id = self.num_tokens + 2 - num_tokens_to_add = 5 - else: - unk_token_id = self.text_tokenizer.convert_token_to_id( - '<|endoftext|>') - cls_token_id = self.text_tokenizer.convert_token_to_id( - '<|endoftext|>') - num_tokens_to_add = 4 - self._command_tokens.extend([ - CommandToken('sop', '<|startofpiece|>', - self.num_tokens), - CommandToken('eop', '<|endofpiece|>', - self.num_tokens + 1), - CommandToken('cls', '[CLS]', cls_token_id), - CommandToken('MASK', - '[MASK]', - self.num_tokens + 3, - lstrip=True), - CommandToken('sep', '[SEP]', self.num_tokens + 4), - CommandToken('unk', '[UNK]', unk_token_id) - ]) - self.num_tokens += num_tokens_to_add - self.num_command_tokens += 6 - - if add_block_symbols: - if add_task_mask: - self._command_tokens.extend([ - CommandToken('gMASK', - '[gMASK]', - self.num_tokens, - lstrip=True), - CommandToken('sMASK', - '[sMASK]', - self.num_tokens + 1, - lstrip=True) - ]) - self.num_tokens += 2 - self.num_command_tokens += 2 - if add_decoder_mask: - self._command_tokens.extend( - [CommandToken('dBLOCK', '[dBLOCK]', self.num_tokens)]) - self.num_tokens += 1 - self.num_command_tokens += 1 - - elif self.tokenizer_class == "sp": - self.num_command_tokens = 0 - self.num_text_tokens = self.text_tokenizer.vocab_size - self.num_tokens = self.num_text_tokens - - if self.tokenizer_model_name.lower().startswith('glm'): - pad_token_id = self.num_tokens - eos_token_id = self.num_tokens - unk_token_id = self.num_tokens + 4 - else: - pad_token_id = self.text_tokenizer.convert_token_to_id('') - eos_token_id = self.text_tokenizer.convert_token_to_id('') - unk_token_id = self.text_tokenizer.convert_token_to_id('') - self._command_tokens = [ - CommandToken('pad', '<|endoftext|>', self.num_text_tokens), - CommandToken('eos', '<|endoftext|>', self.num_text_tokens), - CommandToken('sep', '[SEP]', self.num_text_tokens + 1), - CommandToken('cls', '[CLS]', self.num_text_tokens + 2), - CommandToken('MASK', - '[MASK]', - self.num_text_tokens + 3, - lstrip=True), - CommandToken('unk', '[UNK]', self.num_text_tokens + 4) - ] - - self.num_tokens += 5 - self.num_command_tokens += 6 - self.token_end_id = self.text_tokenizer.convert_token_to_id( - '') - if add_block_symbols: - sop_id = self.text_tokenizer.convert_token_to_id('<|startofpiece|>') - eop_id = self.text_tokenizer.convert_token_to_id('<|endofpiece|>') self._command_tokens.extend([ - CommandToken('sop', '<|startofpiece|>', - self.num_tokens + 1), - CommandToken('eop', '<|endofpiece|>', self.num_tokens + 2) + CommandToken('sop', '<|startofpiece|>', self.num_tokens), + CommandToken('eop', '<|endofpiece|>', self.num_tokens + 1), + CommandToken('cls', '[CLS]', self.num_tokens + 2), + CommandToken('mask', + '[MASK]', + self.num_tokens + 3, + lstrip=True), + CommandToken('sep', '[SEP]', self.num_tokens + 4), + CommandToken('unk', '[UNK]', self.num_tokens + 5) ]) - if fix_command_token: - self.num_tokens += 3 - else: - self.num_tokens += 2 - self.num_command_tokens += 2 + self.num_tokens += 6 + if add_block_symbols: + if not self.tokenizer_class == "bpe": + self.add_command_token('sop', '<|startofpiece|>',self.tokenizer_class) + self.add_command_token('eop', '<|endofpiece|>',self.tokenizer_class) if add_task_mask: if fix_command_token: - self._command_tokens.extend([ - CommandToken('sMASK', - '[sMASK]', - self.num_tokens, - lstrip=True), - CommandToken('gMASK', - '[gMASK]', - self.num_tokens + 1, - lstrip=True) - ]) + self.add_command_token('sMASK', '[sMASK]',self.tokenizer_class) + self.add_command_token('gMASK', '[gMASK]',self.tokenizer_class) else: - self._command_tokens.extend([ - CommandToken('gMASK', - '[gMASK]', - self.num_tokens, - lstrip=True), - CommandToken('sMASK', - '[sMASK]', - self.num_tokens + 1, - lstrip=True) - ]) - self.num_tokens += 2 - self.num_command_tokens += 2 + self.add_command_token('gMASK', '[gMASK]',self.tokenizer_class) + self.add_command_token('sMASK', '[sMASK]',self.tokenizer_class) if add_decoder_mask: - self._command_tokens.extend( - [CommandToken('dBLOCK', '[dBLOCK]', self.num_tokens)]) - self.num_tokens += 1 - self.num_command_tokens += 1 + self.add_command_token('dBLOCK', '[dBLOCK]',self.tokenizer_class) + if add_sentinel_token > 0: + for i in range(1, add_sentinel_token): + self.add_command_token(f'MASK{i}', f'[MASK{i}]',self.tokenizer_class) + self.add_command_token(f'sop{i}', f'<|startofpiece{i}|>',self.tokenizer_class) self.command_name_map = {tok.name: tok for tok in self._command_tokens} self.command_token_map = { tok.token: tok @@ -348,7 +161,17 @@ def __init__(self, } self.command_id_map = {tok.Id: tok for tok in self._command_tokens} self._command_token_tokens = list(self.command_token_map.keys()) - logger.info("All special tokens: %s", str([(k,v.Id) for k,v in self.command_name_map.items()])) + vocab = self.text_tokenizer.get_vocab() + self.token_start_id = vocab.get('', None) + if not self.token_start_id: + self.token_start_id = vocab.get('[CLS]', None) + + self.token_end_id = vocab.get('', None) + if not self.token_end_id: + self.token_end_id = vocab.get('<|endoftext|>', None) + if not self.token_end_id: + self.token_end_id = vocab.get('[SEP]', None) + print("All special tokens: ", str([(k, v.token, v.Id) for k,v in self.command_name_map.items()])) def get_vocab(self): return self.text_tokenizer.get_vocab() @@ -357,9 +180,12 @@ def get_command_id(self, name): """get command token corresponding to `name`""" return self.command_name_map[name].Id - def add_command_token(self, name, token): + def add_command_token(self, name, token, tokenizer_class="wp"): try: - id = self.text_tokenizer.convert_token_to_id(token) + if tokenizer_class == "sp": + id = self.text_tokenizer.get_vocab()[token] + else: + id = self.text_tokenizer.convert_token_to_id(token) except KeyError: id = self.num_tokens self.num_tokens += 1 @@ -459,7 +285,7 @@ def TokenToId(self, token): def DecodeIds(self, ids): """converts ids to wordpiece tokens and joins them as a text string""" - tokens = [] + tokens = [] for id in ids: if id in self.command_id_map: tokens.append(self.command_id_map[id].token) @@ -473,10 +299,14 @@ def DecodeIds(self, ids): tokens, self.command_token_map) def encode(self, text): + if hasattr(self.text_tokenizer, "encode"): + return self.text_tokenizer.encode(text) return self.convert_tokens_to_ids( self.text_tokenizer.tokenize(text)) def decode(self, ids): + if hasattr(self.text_tokenizer, "decode"): + return self.text_tokenizer.decode(ids) return self.DecodeIds(ids) def DecodeTokens(self, tokens): @@ -567,8 +397,8 @@ def encode_plus_non_glm( ): def get_input_ids(text): - tokens = self.text_tokenizer.tokenize(text) - return self.text_tokenizer.convert_tokens_to_ids(tokens) + tokens = self.tokenize(text) + return self.convert_tokens_to_ids(tokens) first_ids = get_input_ids(text) second_ids = get_input_ids( @@ -636,10 +466,16 @@ def encode_plus( # for Seq2seq max_length=None, padding=True, ): - if not self.tokenizer_model_name.lower().startswith("glm") and not self.tokenizer_model_name.lower().startswith( + if hasattr(self.text_tokenizer, "encode_plus"): + return self.text_tokenizer.encode_plus(source_text) + elif not self.tokenizer_model_name.lower().startswith("glm") and not self.tokenizer_model_name.lower().startswith( "alm"): return self.encode_plus_non_glm(source_text, second_text, truncation, max_length) + + + # elif self.tokenizer_model_name.lower().startswith("opt"): + # return None sop_id = self.get_command_id('sop') # start of piece eop_id = self.get_command_id('eop') # end of piece sep_id = self.get_command_id('sep') # seperation @@ -709,14 +545,12 @@ def truncate_sequence(max_length, def tokenize_as_tensor(self, texts): """ Returns the tokenized representation of given input string(s) - Parameters ---------- texts : Union[str, List[str]] An input string or a list of input strings to tokenize context_length : int The context length to use; all CLIP models use 77 as the context length - Returns ------- A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length] @@ -737,4 +571,42 @@ def tokenize(self, text, maxlen=None, add_spatial_tokens=False): if maxlen is not None: index = int(self.get_command_id('sep') is not None) + 1 self.truncate_sequence(maxlen, tokens, pop_index=-index) - return tokens \ No newline at end of file + return tokens + + def search_special(self, name): + if name == "cls": + if self.check_special(''): return '' + elif self.check_special('[CLS]'): return '[CLS]' + elif name == "pad": + if self.check_special(''): return '' + elif self.check_special('[PAD]'): return '[PAD]' + elif self.check_special('<|endoftext|>'): return '<|endoftext|>' + elif name == "eos": + if self.check_special(''): return '' + elif self.check_special('|endoftext|'): return '|endoftext|' + elif self.check_special('[PAD]'): return '[PAD]' + elif name == "sep": + if self.check_special(''): return '' + elif self.check_special('[SEP]'): return '[SEP]' + elif name == "unk": + if self.check_special(''): return '' + elif self.check_special('[UNK]'): return '[UNK]' + elif name == "bos": + if self.check_special(''): return '' + elif name == "mask": + if self.check_special('[MASK]'): return '[MASK]' + elif self.check_special(''): return '' + elif name == "eod": + if self.check_special(''): return '' + return None + + def check_special(self, tk): + + try: + if self.tokenizer_class == 'sp': + self.text_tokenizer.get_vocab()[tk] + else: + self.text_tokenizer.convert_token_to_id(tk) + return True + except KeyError: + return False diff --git a/flagai/env_trainer.py b/flagai/env_trainer.py index 8710da1f..a1bf81c7 100644 --- a/flagai/env_trainer.py +++ b/flagai/env_trainer.py @@ -209,18 +209,15 @@ def get_dataloader(self, dataset, collate_fn, shuffle=False): shuffle=shuffle) else: if self.env_type == 'deepspeed+mpu': - # num_replicas = self.world_size // mpu.get_model_parallel_world_size( - # ) - # rank = self.rank // mpu.get_model_parallel_world_size() - # rank = mpu.get_model_parallel_rank() rank = mpu.get_model_parallel_src_rank() - print("*"*80) - print("local rank",self.rank, "model rank", rank) - print("*"*80) + data_rank = mpu.get_data_parallel_rank() + log_dist("*"*80) + log_dist(f"local rank {self.rank} src rank {rank} data rank {data_rank}") + log_dist("*"*80) sampler = torch.utils.data.distributed.DistributedSampler( dataset, - # num_replicas=num_replicas, - rank=rank, + num_replicas=self.world_size//self.model_parallel_size, + rank=data_rank, shuffle=shuffle) elif self.env_type == 'bmtrain': print("*"*80) diff --git a/flagai/model/base_model.py b/flagai/model/base_model.py index 46367e28..c385c52a 100644 --- a/flagai/model/base_model.py +++ b/flagai/model/base_model.py @@ -77,9 +77,11 @@ def from_pretrain(cls, config_path = os.path.join(download_path, "config.json") checkpoint_path = os.path.join(download_path, "pytorch_model.bin") - def load_local(checkpoint_path): + def load_local(checkpoint_path, only_download_config=False): model = cls.init_from_json(config_path, **kwargs) model.to(device) + if only_download_config: + return model if os.getenv('ENV_TYPE') != 'deepspeed+mpu': if os.path.exists(checkpoint_path): model.load_weights(checkpoint_path) @@ -146,7 +148,7 @@ def load_diffusion_local(yaml_path, only_download_config=False, **kwargs): It is fine when checkpoint_path does not exist, for the case that only_download_config=True At that time the model will not be loaded. """ - return load_local(checkpoint_path) + return load_local(checkpoint_path, only_download_config=only_download_config) try: model_id = _get_model_id(model_name) diff --git a/flagai/model/gpt2_model.py b/flagai/model/gpt2_model.py index af8aa156..2f9a63bf 100644 --- a/flagai/model/gpt2_model.py +++ b/flagai/model/gpt2_model.py @@ -10,7 +10,6 @@ from flagai.model.utils import normal_init_method from flagai.model.base_model import BaseModel import torch.nn.functional as F - if os.getenv('ENV_TYPE') == 'deepspeed+mpu': from flagai.mpu.utils import divide from flagai.mpu.random import checkpoint @@ -124,6 +123,7 @@ def __init__(self, config): GPT2Block(config.n_ctx, config, scale=True) for _ in range(config.n_layer) ]) + self.ln_f = nn.LayerNorm(config.n_embd, eps=config.layer_norm_epsilon) self.device_map = None @@ -301,6 +301,7 @@ def __init__(self, config, **kwargs): bias=False) init_method(self.lm_head.weight) + def _make_causal_mask(self, input_ids): device = input_ids.device bsz, tgt_len = input_ids.shape diff --git a/flagai/model/predictor/gpt.py b/flagai/model/predictor/gpt.py index bab9004a..156dd4f0 100644 --- a/flagai/model/predictor/gpt.py +++ b/flagai/model/predictor/gpt.py @@ -7,10 +7,10 @@ def gpt_random_sample_use_cache(model, tokenizer, text, input_max_length, out_ma top_k, top_p, repetition_penalty, temperature, device): tokenizer_out = tokenizer.encode_plus(text, max_length=input_max_length) token_ids = tokenizer_out["input_ids"] + token_end_id = tokenizer.get_command_id('sep') token_eos_id = tokenizer.get_command_id('eos') removed_tokens = [token_end_id, token_eos_id] - while len(token_ids)>0 and token_ids[-1] in removed_tokens: token_ids = token_ids[:-1] diff --git a/flagai/model/predictor/utils.py b/flagai/model/predictor/utils.py index da7eec51..a39d3d2b 100644 --- a/flagai/model/predictor/utils.py +++ b/flagai/model/predictor/utils.py @@ -1117,7 +1117,6 @@ def alm_beamsearch(model, tokenizer, text, out_max_length, beam_size, eod_token= context_length = context_length_tensor[0].item() context_tokens_tensor = torch.LongTensor(context_tokens) text = tokenizer.DecodeIds(context_tokens_tensor.tolist()) - start_time = time.time() mems = [] tokens = context_tokens_tensor @@ -1134,7 +1133,7 @@ def alm_beamsearch(model, tokenizer, text, out_max_length, beam_size, eod_token= dtype=torch.long) position_ids = torch.stack((position_ids, block_position_ids), dim=0) position_ids = position_ids.unsqueeze(0) - mask_tokens = ['MASK', 'sMASK', 'gMASK'] + mask_tokens = ['mask', 'sMASK', 'gMASK'] mask_tokens = [tokenizer.get_command_id(token) for token in mask_tokens] end_tokens = [tokenizer.get_command_id('eop'), eod_token] mask_positions = [] @@ -1426,7 +1425,7 @@ def glm_generate_sample( context_length = context_length_tensor[0].item() context_tokens_tensor = torch.LongTensor(context_tokens) text = tokenizer.DecodeIds(context_tokens_tensor.tolist()) - + start_time = time.time() mems = [] tokens = context_tokens_tensor @@ -1443,7 +1442,7 @@ def glm_generate_sample( dtype=torch.long) position_ids = torch.stack((position_ids, block_position_ids), dim=0) position_ids = position_ids.unsqueeze(0) - mask_tokens = ['MASK', 'sMASK', 'gMASK'] + mask_tokens = ['mask', 'sMASK', 'gMASK'] mask_tokens = [tokenizer.get_command_id(token) for token in mask_tokens] end_tokens = [tokenizer.get_command_id('eop'), eod_token] mask_positions = [] diff --git a/flagai/mp_tools.py b/flagai/mp_tools.py index b4e9a108..f4dfff0b 100644 --- a/flagai/mp_tools.py +++ b/flagai/mp_tools.py @@ -412,4 +412,4 @@ def change_pytorch_model_mp_from_n_to_1(model_name_brief, checkpoint): if __name__ == "__main__": change_pytorch_model_mp_from_1_to_n( - '/mnt/test_10b_models/state_dict/GLM-10b-en', 2) + '/mnt/test_10b_models/state_dict/GLM-10b-en', 2) \ No newline at end of file diff --git a/flagai/optimizers.py b/flagai/optimizers.py index 43be0138..8d0867b3 100644 --- a/flagai/optimizers.py +++ b/flagai/optimizers.py @@ -103,6 +103,34 @@ def get_optimizer(param_groups, lr=lr, relative_step=False, warmup_init=False) + elif optimizer == 'adamw': + from torch.optim import AdamW + optimizer = AdamW(param_groups, + lr=lr, + weight_decay=weight_decay, + betas=(adam_beta1, adam_beta2), + eps=adam_eps) + elif optimizer == 'lion': + from lion_pytorch import Lion + optimizer = Lion(param_groups, + lr=lr, + weight_decay=weight_decay, + betas=(adam_beta1, adam_beta2) + ) + elif optimizer == 'adan': + from adan import Adan + optimizer = Adan(param_groups, + lr=lr, + weight_decay=weight_decay, + betas=(adam_beta1, adam_beta2, 0.99), + eps=adam_eps) + elif optimizer == 'lamb': + from torch_optimizer import Lamb + optimizer = Lamb(param_groups, + lr=lr, + weight_decay=weight_decay, + betas=(adam_beta1, adam_beta2), + eps=adam_eps) else: raise NotImplementedError diff --git a/flagai/test_utils.py b/flagai/test_utils.py index 83dacde3..5faa0aec 100644 --- a/flagai/test_utils.py +++ b/flagai/test_utils.py @@ -14,7 +14,7 @@ def build_input_from_ids(text_a_ids=None, mask_id=None, masked_lm=False): if mask_id is None: - mask_id = tokenizer.get_command_id('MASK') + mask_id = tokenizer.get_command_id('mask') eos_id = tokenizer.get_command_id('eos') cls_id = tokenizer.get_command_id('cls') sep_id = tokenizer.get_command_id('sep') diff --git a/flagai/trainer.py b/flagai/trainer.py index 709cd7c6..680f177b 100644 --- a/flagai/trainer.py +++ b/flagai/trainer.py @@ -162,6 +162,7 @@ def __init__( deepspeed_config=None, model_parallel_size=1, training_script="train.py", + optimizer_type='adam', ): if timers is not None: @@ -188,6 +189,8 @@ def __init__( self.eval_interval = eval_interval self.tokenizer = tokenizer + self.optimizer_type = optimizer_type + # model checkpointing self.save_dir = save_dir self.save_interval = save_interval @@ -340,13 +343,14 @@ def get_dataloader(self, dataset, collate_fn, shuffle=False): # rank = self.rank // mpu.get_model_parallel_world_size() # rank = mpu.get_model_parallel_rank() rank = mpu.get_model_parallel_src_rank() - print("*"*80) - print("local rank",self.rank, "model rank", rank) - print("*"*80) + data_rank = mpu.get_data_parallel_rank() + log_dist("*"*80) + log_dist(f"local rank {self.rank} src rank {rank} data rank {data_rank}") + log_dist("*"*80) sampler = torch.utils.data.distributed.DistributedSampler( dataset, - # num_replicas=num_replicas, - rank=rank, + num_replicas=self.world_size//self.model_parallel_size, + rank=data_rank, shuffle=shuffle) elif self.env_type == 'bmtrain': print("*"*80) @@ -491,6 +495,7 @@ def train(self, optimizer='adam') # if not self.fp16 else 'adafactor') self.total_iter = int(self.epochs * len(train_dataloader)) + if lr_scheduler == None and optimizer != None and self.warm_up > 0 and 'deepspeed' not in self.env_type and self.epochs > 0: if self.env_type == 'bmtrain': ## lr_scheduler.step with optim_manager.step @@ -1067,7 +1072,6 @@ def evaluate(self, labels = data_iterator['labels'] else: labels = data_iterator['target_ids'] - loss_mask = data_iterator['loss_mask'] if len(self.metric_methods) != 0: if {metric_tuple[0] for metric_tuple in self.metric_methods} & {"rouge", "bleu"}: batch_preds = torch.argmax(logits.detach(), dim=-1).cpu() diff --git a/setup.py b/setup.py index be63eb7d..beaf4f63 100644 --- a/setup.py +++ b/setup.py @@ -5,7 +5,7 @@ setup( name="flagai", - version="v1.5.1", + version="v1.6.1", description="FlagAI aims to help researchers and developers to freely train and test large-scale models for NLP/CV/VL tasks.", long_description=open("README.md", encoding="utf-8").read(), long_description_content_type="text/markdown", diff --git a/test.py b/test.py deleted file mode 100644 index fce506e0..00000000 --- a/test.py +++ /dev/null @@ -1,13 +0,0 @@ -# Copyright © 2022 BAAI. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License") -import unittest - -print('test syn') -test_dir = './tests' -test_report_path = './test_report' -discover = unittest.defaultTestLoader.discover(test_dir, pattern='test_*.py') -with open(test_report_path, "w") as report_file: - runner = unittest.TextTestRunner(stream=report_file, verbosity=2) - #runner=unittest.TextTestRunner() - runner.run(discover) \ No newline at end of file diff --git a/tests/test_tokenizer.py b/tests/test_tokenizer.py index ffc6de0e..c72d34a7 100644 --- a/tests/test_tokenizer.py +++ b/tests/test_tokenizer.py @@ -14,6 +14,14 @@ def test_tokenizer_GLM_large_ch(self): [3378, 1567, 2613, 20282], 'EncodeAsIds Error') self.assertEqual(tokenizer.DecodeIds([3378, 1567, 2613, 20282]), '今天吃饭吃了肯德基', 'DecodeIds Error') + self.assertEqual(tokenizer.tokenize('今天吃饭吃了肯德基'), + ['▁今天', '吃饭', '吃了', '肯德基'], 'tokenize Error') + self.assertEqual(tokenizer.encode_plus('今天吃饭吃了肯德基')['input_ids'], + [50006, 3378, 1567, 2613, 20282, 50001], 'encode_plus Error') + self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + {('pad', '<|endoftext|>', 50000), ('eos', '<|endoftext|>', 50000), ('sep', '[SEP]', 50001), + ('cls', '[CLS]', 50002), ('mask', '[MASK]', 50003), ('unk', '[UNK]', 50004), ('sop', '<|startofpiece|>', 50006), + ('eop', '<|endofpiece|>', 50007), ('sMASK', '[sMASK]', 50008), ('gMASK', '[gMASK]', 50009)}, 'SpecialTokens error') def test_tokenizer_GLM_large_en(self): tokenizer = Tokenizer.from_pretrained("GLM-large-en") @@ -22,6 +30,10 @@ def test_tokenizer_GLM_large_en(self): [13017, 7975, 3084, 2033, 3407], '') self.assertEqual(tokenizer.DecodeIds([13017, 7975, 3084, 2033, 3407]), 'fried chicken makes me happy', 'DecodeIds Error') + self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + {('eos', '[PAD]', 0), ('cls', '[CLS]', 101), ('mask', '[MASK]', 103), ('unk', '[UNK]', 100), + ('sep', '[SEP]', 102), ('pad', '[PAD]', 0), ('sop', '<|startofpiece|>', 30522), ('eop', '<|endofpiece|>', 30523), + ('gMASK', '[gMASK]', 30524), ('sMASK', '[sMASK]', 30525)}) # def test_tokenizer_glm_10b_en(self): # tokenizer = Tokenizer.from_pretrained("GLM-10b-en") @@ -30,23 +42,43 @@ def test_tokenizer_GLM_large_en(self): # [25520, 9015, 1838, 502, 3772], '') # self.assertEqual(tokenizer.DecodeIds([25520, 9015, 1838, 502, 3772]), # 'fried chicken makes me happy', 'DecodeIds Error') + # self.assertEqual([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()], + # [('eos', '[PAD]', 0), ('cls', '[CLS]', 101), ('mask', '[MASK]', 103), ('unk', '[UNK]', 100), + # ('sep', '[SEP]', 102), ('pad', '[PAD]', 0), ('sop', '<|startofpiece|>', 30522), ('eop', '<|endofpiece|>', 30523), + # ('gMASK', '[gMASK]', 30524), ('sMASK', '[sMASK]', 30525)]) + def test_tokenizer_t5(self): - tokenizer = Tokenizer.from_pretrained('t5-base-en') - self.assertEqual(tokenizer.TokenToId("day"), 1135, '') - self.assertEqual(tokenizer.EncodeAsIds("fried chicken makes me happy"), - [3, 7704, 3832, 656, 140, 1095], '') - self.assertEqual(tokenizer.DecodeIds([3, 7704, 3832, 656, 140, 1095]), - 'fried chicken makes me happy', 'DecodeIds Error') + tokenizer = Tokenizer.from_pretrained('T5-base-ch') + self.assertEqual(tokenizer.TokenToId("人"), 297, '') + self.assertEqual(tokenizer.EncodeAsIds("今天吃饭吃了肯德基"), + [306, 1231, 798, 5447, 798, 266, 4017, 1738, 1166], '') + self.assertEqual(tokenizer.DecodeIds([306, 1231, 798, 5447, 798, 266, 4017, 1738, 1166]), + '今天吃饭吃了肯德基', 'DecodeIds Error') + encode_plus_result = tokenizer.encode_plus("今天吃饭吃了肯德基") + self.assertEqual(list(encode_plus_result.keys()), + ['input_ids', 'token_type_ids'], 'encode_plus Error') + self.assertEqual(encode_plus_result['input_ids'], + [101, 306, 1231, 798, 5447, 798, 266, 4017, 1738, 1166, 102], 'encode_plus Error') + self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + {('eos', '[PAD]', 0), ('cls', '[CLS]', 101), ('mask', '[MASK]', 103), ('unk', '[UNK]', 100), + ('sep', '[SEP]', 102), ('pad', '[PAD]', 0)}, 'SpecialTokens error') + def test_tokenizer_roberta(self): tokenizer = Tokenizer.from_pretrained('RoBERTa-base-ch') - # print(tokenizer.DecodeIds([791, 1921, 1391, 7649, 1391, 749, 5507, 2548, 1825])) self.assertEqual(tokenizer.TokenToId("人"), 782, '') self.assertEqual(tokenizer.EncodeAsIds("今天吃饭吃了肯德基"), [791, 1921, 1391, 7649, 1391, 749, 5507, 2548, 1825], '') self.assertEqual(tokenizer.DecodeIds([791, 1921, 1391, 7649, 1391, 749, 5507, 2548, 1825]), '今天吃饭吃了肯德基', 'DecodeIds Error') + self.assertEqual(tokenizer.tokenize('今天吃饭吃了肯德基'), + ['今', '天', '吃', '饭', '吃', '了', '肯', '德', '基'], 'tokenize Error') + self.assertEqual(tokenizer.encode_plus('今天吃饭吃了肯德基')['input_ids'], + [101, 791, 1921, 1391, 7649, 1391, 749, 5507, 2548, 1825, 102], 'encode_plus Error') + self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + {('unk', '[UNK]', 100), ('cls', '[CLS]', 101), ('sep', '[SEP]', 102), ('mask', '[MASK]', 103), + ('eos', '[PAD]', 0), ('pad', '[PAD]', 0)}, 'SpecialTokens error') def test_tokenizer_bert(self): tokenizer = Tokenizer.from_pretrained('BERT-base-en') @@ -55,26 +87,48 @@ def test_tokenizer_bert(self): [13017, 7975, 3084, 2033, 3407], '') self.assertEqual(tokenizer.DecodeIds([13017, 7975, 3084, 2033, 3407]), 'fried chicken makes me happy', 'DecodeIds Error') + self.assertEqual(tokenizer.tokenize('fried chicken makes me happy'), + ['fried', 'chicken', 'makes', 'me', 'happy'], 'tokenize Error') + self.assertEqual(tokenizer.encode_plus('fried chicken makes me happy')['input_ids'], + [101, 13017, 7975, 3084, 2033, 3407, 102], 'encode_plus Error') + self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + {('eos', '[PAD]', 0), ('unk', '[UNK]', 100), ('cls', '[CLS]', 101), ('sep', '[SEP]', 102), + ('mask', '[MASK]', 103), ('pad', '[PAD]', 0)}, 'SpecialTokens error') - def test_tokenizer_cpm1(self): - loader = AutoLoader(task_name="lm", - model_name="CPM-large-ch", - model_dir="./checkpoints/", - only_download_config=True) - tokenizer = loader.get_tokenizer() - self.assertEqual(tokenizer.encode("day"), [8, 8275], '') - self.assertEqual(tokenizer.encode("fried chicken makes me happy"), - [2487, 27385, 9291, 9412, 3531, 14588, 289, 4406, 25239], '') - self.assertEqual(tokenizer.decode([2487, 27385, 9291, 9412, 3531, 14588, 289, 4406, 25239]), - 'fried chicken makes me happy', 'DecodeIds Error') + # def test_tokenizer_cpm1(self): + # loader = AutoLoader(task_name="lm", + # model_name="CPM-large-ch", + # model_dir="./checkpoints/", + # only_download_config=True) + + # tokenizer = loader.get_tokenizer() + # self.assertEqual(tokenizer.TokenToId("人"), 62, '') + # self.assertEqual(tokenizer.encode("今天吃饭吃了肯德基"), + # [837, 3079, 1777, 3079, 139, 3687, 513, 1463], '') + # self.assertEqual(tokenizer.DecodeIds([837, 3079, 1777, 3079, 139, 3687, 513, 1463]), + # '今天吃饭吃了肯德基', 'DecodeIds Error') + # self.assertEqual(tokenizer.tokenize('今天吃饭吃了肯德基'), + # [837, 3079, 1777, 3079, 139, 3687, 513, 1463], 'tokenize Error') + # self.assertEqual(tokenizer.encode_plus('今天吃饭吃了肯德基')['input_ids'], + # [837, 3079, 1777, 3079, 139, 3687, 513, 1463], 'encode_plus Error') + # self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + # {('unk', '', 0), ('cls', '', 1), ('eos', '', 2), ('sep', '', 4), + # ('mask', '', 6), ('pad', '', 5),('eod', '', 7)}, 'SpecialTokens error') def test_tokenizer_opt(self): - tokenizer = Tokenizer.from_pretrained('opt-125m-en') + tokenizer = Tokenizer.from_pretrained('opt-1.3b-en') self.assertEqual(tokenizer.encode("day"), [1208], '') self.assertEqual(tokenizer.encode_plus("fried chicken makes me happy")["input_ids"], - [50260, 21209, 5884, 817, 162, 1372, 50260], '') + [0, 21209, 5884, 817, 162, 1372, 2], '') self.assertEqual(tokenizer.decode([21209, 5884, 817, 162, 1372]), 'fried chicken makes me happy', 'DecodeIds Error') + self.assertEqual(tokenizer.tokenize('fried chicken makes me happy'), + ['fried', 'Ġchicken', 'Ġmakes', 'Ġme', 'Ġhappy'], 'tokenize Error') + self.assertEqual(tokenizer.encode_plus('fried chicken makes me happy')['input_ids'], + [0, 21209, 5884, 817, 162, 1372, 2], 'encode_plus Error') + self.assertEqual(set([(k, v.token, v.Id) for k,v in tokenizer.command_name_map.items()]), + {('cls', '', 0), ('pad', '', 1), ('bos', '', 2), ('eos', '', 2), ('unk', '', 3), + ('mask', '', 50264)}, 'SpecialTokens error') def test_tokenizer_clip(self): loader = AutoLoader(task_name="txt_img_matching", @@ -89,6 +143,7 @@ def test_tokenizer_evaclip(self): self.assertEqual(tokenizer.tokenize_as_tensor("cat")[0][:3].tolist(), [49406, 2368, 49407], '') + def suite(): suite = unittest.TestSuite() suite.addTest(TokenizerTestCase('test_tokenizer_GLM_large_ch')) @@ -97,7 +152,7 @@ def suite(): suite.addTest(TokenizerTestCase('test_tokenizer_t5')) suite.addTest(TokenizerTestCase('test_tokenizer_roberta')) suite.addTest(TokenizerTestCase('test_tokenizer_bert')) - suite.addTest(TokenizerTestCase('test_tokenizer_cpm1')) + # suite.addTest(TokenizerTestCase('test_tokenizer_cpm1')) suite.addTest(TokenizerTestCase('test_tokenizer_opt')) suite.addTest(TokenizerTestCase('test_tokenizer_clip')) suite.addTest(TokenizerTestCase('test_tokenizer_evaclip')) @@ -107,4 +162,4 @@ def suite(): if __name__ == '__main__': runner = unittest.TextTestRunner() - runner.run(suite()) + runner.run(suite()) \ No newline at end of file