Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[prompt] add doc #3362

Merged
merged 13 commits into from
Oct 10, 2022
Merged

[prompt] add doc #3362

merged 13 commits into from
Oct 10, 2022

Conversation

LemonNoel
Copy link
Contributor

@LemonNoel LemonNoel commented Sep 23, 2022 •
edited
Loading

PR types

Others

PR changes

Docs

Description

  • Add document for prompt API

截屏2022-09-28 16 20 00

@LemonNoel LemonNoel self-assigned this Sep 23, 2022
@LemonNoel LemonNoel requested a review from ZHUI September 27, 2022 12:14
ZHUI
ZHUI reviewed Sep 27, 2022
View reviewed changes
docs/advanced_guide/prompt.rst Outdated Show resolved Hide resolved
docs/advanced_guide/prompt.rst Outdated
- ``hard`` ：表示手工构造的提示。
- ``mask`` ：表示待预测的 token。
- ``sep`` 表示在句间增加 ``[SEP]`` 区分不同的句子。
- ``soft`` 表示连续向量。若值为 ``None`` ，则使用随机初始化；若值为文本，则使用预训练词向量中对应的向量进行初始化。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soft 这里可以为多个字的向量初始化吗？

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

还是再hard的基础上，叠加一个小的soft可学习向量？

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soft 可以为多个字初始化向量，定义几个字，就是几个 soft 向量。

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在模板定义示例中加上具体的文字说明。

docs/advanced_guide/prompt.rst Outdated

"{'soft': None}{'text': 'text_a'}{'sep'}{'text': 'text_b'}"

- 使用 ``duplicate`` 参数快速定义多个提示向量：
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以叠加任意字段，还是只有soft？

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

只有 soft 可以使用，并且仅限随机初始化的 soft token。

docs/advanced_guide/prompt.rst Outdated Show resolved Hide resolved
docs/advanced_guide/prompt.rst Outdated Show resolved Hide resolved
docs/advanced_guide/prompt.rst Outdated Show resolved Hide resolved
docs/advanced_guide/prompt.rst Outdated
- ``tokenizer`` : 预训练模型的 tokenizer，用于预测词的编码。
- ``labels`` : 数据集的原标签列表（可选）。
- ``label_words`` : 原标签到预测词之间的映射字典。如果同时定义了 ``labels`` ，二者的标签集合需要相同。
- ``prefix`` : 预测词解码前增加的前缀，用于 ``RoBERTa`` 等对前缀敏感的模型。默认为 ``None`` ，无前缀。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不明白

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对于 RoBERTa tokenizer 来说，词前加空格和不加空格，比如' yes' 和 'yes' 的编码结果是不一样的。文档里补上例子。

@LemonNoel LemonNoel requested a review from ZHUI September 28, 2022 11:54
ZHUI
ZHUI previously approved these changes Oct 10, 2022
View reviewed changes
Copy link
Collaborator

@ZHUI ZHUI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

docs/advanced_guide/prompt.md
【提示学习】通过提示语句和标签词映射的定义，转化为 MLM 任务，无需训练新的参数，适用于小样本场景。


Prompt API 提供了这类算法实现的基本模块，支持[PET](https://arxiv.org/abs/2001.07676)、[P-Tuning](https://arxiv.org/abs/2103.10385)、[WARP](https://aclanthology.org/2021.acl-long.381/)、[RGL](https://aclanthology.org/2022.findings-naacl.81/)等经典算法的快速实现。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这块论文，可以最后放一个参考文献

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已增加参考文献。

docs/advanced_guide/prompt.md
* [数据准备](#数据准备)
* [预训练参数准备](#预训练参数准备)
* [定义提示学习模型](#定义提示学习模型)
* [使用PromptTrainer训练](#使用PromptTrainer训练)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用PromptTrainer训练 链接好像失效了

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我这边试了下可以跳转的。

ZHUI
ZHUI reviewed Oct 10, 2022
View reviewed changes
docs/advanced_guide/prompt.md Outdated

```python

import paddle
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里代码前面全部有换行。

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已删除缩进。

@ZHUI
Copy link
Collaborator

ZHUI commented Oct 10, 2022

代码示例部分，要保证 用户 复制下来可以直接跑。
分散成多部分，多段的代码，保证用户合并在一起后可以直接跑。cc: @LemonNoel

@LemonNoel
Copy link
Contributor Author

代码示例部分，要保证 用户 复制下来可以直接跑。 分散成多部分，多段的代码，保证用户合并在一起后可以直接跑。cc: @LemonNoel

已测试，demo部分均可正常跑通。

@LemonNoel LemonNoel requested a review from ZHUI October 10, 2022 10:01
ZHUI
ZHUI approved these changes Oct 10, 2022
View reviewed changes
Copy link
Collaborator

@ZHUI ZHUI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LemonNoel LemonNoel merged commit b6a718b into PaddlePaddle:develop Oct 10, 2022
@LemonNoel LemonNoel mentioned this pull request Oct 13, 2022
Closed
linjieccc added a commit that referenced this pull request Oct 17, 2022
@linjieccc @wj-Mcat
@JunnYu @w5688414 @JamesLim-sy @chenxiaozeng @guoshengCS @bruce0210 @wuhuachaocoding @ZHUI @wawltor @FrostML @kztao @joey12300 @paopjian @lugimzzz @LiuChiachi @0x45f @HexToString @ZeyuChen @zhengya01 @sljlp @LemonNoel @WenmuZhou @Liujie0926 @westfish @douseful @Zhiyuan-Fan @YanhuiDua @haohongxiang @Yam0214 @sneaxiy @alkaideemo @ShawnNew @qipengh @sijunhe
Add ERNIE-Layout to release 2.4 branch (#3470)
4ccf80d 
* fix multi-layer-inherit

* update bert model unittest

* update requirements.txt

* update ernie modeling test

* update roberta unittest

* update roformer modeling testing

* complete ernie label loss

* complete ernie/roberta/roformer unittest

* update label/loss

* update refactor code

* remove unrelated requirements

* add license

* Update setup.py and README Examples (#3208)

* Move token_num fetch out of train cycle (#3089)

* Add finance course (#3207)

* add finance course group code

Co-authored-by: tianxin <[email protected]>

* Update README_cn.md (#3212)

add v2.4 features description.

* Update README.md (#3209)

Improve and fix the text content of case 1.

Co-authored-by: tianxin <[email protected]>

* [Recompute] Update recompute for hybrid parallel interface. (#3211)

Co-authored-by: Zhong Hui <[email protected]>

* Update README_cn.md

* [ModelingOutput]update roformer unittest (#3159)

* add roformer unittest

* add roformer unittest

* update test_modeling

* use relative import

* reduce model config to accelerate testing

* remove input_embedding from pretrained model

* revert slow tag

* update local branch

* update get_vocab method

* update get_vocab method

* update test_chinese method

* change absolute import

* update unittest

* update chinese test case

* add roformer more output testing

Co-authored-by: Guo Sheng <[email protected]>
Co-authored-by: liu zhengxi <[email protected]>

* Update README_cn.md

* Fix windows dtype bug of neural search (#3182)

* Fix windows dtype bug of neural search

* Fix windows dtype bug of neural search

Co-authored-by: 吴高升 <[email protected]>

* Update README_cn.md

* Update README_cn.md

* Update README_cn.md

* [ModelingOutput]add more output for skep model (#3146)

* update return_dict/label in skep model

* complete skep add-more-output

* refactor simple code

Co-authored-by: Zhong Hui <[email protected]>
Co-authored-by: Guo Sheng <[email protected]>
Co-authored-by: liu zhengxi <[email protected]>

* remove model_config_file and resource_files_names

* Update README_cn.md (#3219)

* Remove boost library. (#3215)

* Remove boost library.

* add conditional include for gtest

* Add test, demo exclude

* Update bos url for UIE (#3222)

* Update bos url

* Update README.md

* Update README.md

* 源码安装htbuilder,避免windows安装失败 (#3221)

Co-authored-by: 吴高升 <[email protected]>

* not default to gpu (#3218)

* Update codegen params and doc (#3228)

* update decoding

* update doc

* update three models

* [Unittest]add roformerv2 unittest (#2994)

* add roformerv2 unittest

* update roformer-v2 testing

* update config to accelerate testing

* remove comment

Co-authored-by: Guo Sheng <[email protected]>

* Optimize text classification deploy  (#3217)

* optimize_deploy

* optimize_deploy

* update_readme

* fix data distill for UIE (#3231)

* fix data distill

* update

* add evaluate_teacher

* [Pre-Training]  ERNIE-CW pre-training tasks docs. (#3111)

* add ernie-large config

* update

* update clue finetune.

* unused delete.

* update

* support no nsp for enrie.

* fix evaluation

* fix amp o2 save_dtype bugs.

* extand ernie.

* fix ernie pretrain with ## vocab.

* extend vocab

* support custom tokenizer.

* add some comments.

* fix bugs.

* add comments.

* fix bug.

* fix run_pretrain_static logging.

* fix all gather.

* fix a100

* fix

* fix bugs

* fix save

* tmp commit for pre-process.

* Update README.md

* Update README.md

* add amp o1 support

* ernie cw readme.

* fix

* throw error when dataset is invalid.

* update document.

* refine readme.

* fix

* refactor

* refator2

* Add pre-training introduction.

* update  image width.

* refine doc

* fit table width.

* fix c++ style

* fix table

* refine docs

* refine model_zoo/ernie-1.0/README.md

* readfine readme.

* fix link

* fix bug

* fix documents.

* add weight.

* fix config

* Update README.md & Add more data into csv& change UI (#3237)

* fix bug of label dimension smaller than 1 (#3238)

* update output dirname of compression api (#3252)

* [ModelingOutput] add tinybert/Electra/XLNet/ALBERT/ERNIE-M more output & loss (#3148)

* complete tinybert more output & loss

* complete tinybert/erniem output

* complete xlnet unittest

* complete the electra unittest

* complete albert more modeling output

* complete albert more modeling output

* complete ernie-doc model more output

* revert ernie-doc modeling

* update more output

* update model testing

* convert paddle.is_tensor -> isinstance

* update tinybert & electra models

* Add unit tests for T5 (#3115)

* analysis_module_bug_fix (#3246)

* [CodeStyle] Add copyright for python file. (#3259)

* Add copyright for python files.

* [IssueTemplate] Add issue template (#3251)

* update issue-template

* remove old issue template

* add id field to template

* update github issue template

* [BugFix]update vocab_size in init_config (#3260)

* update vocab_size in init_config

* make update_init_config more common

Co-authored-by: Zhong Hui <[email protected]>

* update t5 tests (#3266)

* Update debug mode for relation prompt (#3263)

* update debug mode for relation prompt

* update

* update

* Update README.md and Rename dir to FAQ directory (#3272)

* [DOC] Add ernie-1.0-base-zh-cw benchmark results. (#3248)

* [DOC] Update highlights of README.md (#3278)

* Update README.md

* Update README.md

* Add unit tests for UnifiedTransformer (#3177)

* [Trainer] Support recompute for trainer. (#3261)

* support recompute for trainer.

* Upgrade FAQ finance to Milvus 2.1 (#3267)

* Upgrade FAQ finance to Milvus 2.1

* Update text format for faq

* Update feature_extract.sh

* Fix ft substr bug (#3279)

* optimize cmakelist

* Add substr pos check

* remove glog/logging.h (#3280)

* Update ft version to 0.2.0 (#3285)

* update docs wechat code (#3284)

* update link typo (#3236)

* add_dataset_link (#3286)

* Add use_faster flag for uie of taskflow. (#3194)

* Add use_faster flag for taskflow

* Add empty line

* Add doc of uie

* remove faster_tokenizer tmp

* merge

* fix import error (#2853)

* [TIPC]Support @to_static train for base-transformer (#3277)

* [TIPC]Support @to_static train for base-transformer

* Fix to_static args

* Add ft compile doc and scripts (#3292)

* Fix the mac compile

* Add cpp, python lib building scripts

* Remove cache in cpp lib

* Add compile docs

* fix ft build script (#3293)

* Add Milvus2.1 Support and Update pipielines qa ui  (#3283)

* Add Milvus Support and Update pipielines qa ui

* Remove unused comments

* fix bug of relation example is empty (#3295)

* Compression API Supports ERNIE-M and more Pretrained models (#3234)

* update compression doc

* update compression doc

* support more models and update compression api

* update inputspec info, avoid error

* optimize train.py (#3300)

* update ernie task tipc

* update

* optimize_sparse_strategy (#3311)

* Add FAQ and missing json output files (#3298)

* Add Docker compile Support for Pipelines (#3315)

* Add Docker compile Support

* change cuda to uppercase

* Update README_en.md (#3320)

* Update README_en.md

* Update README_en.md

* Update README_en.md

* Update README_en.md

* Update README_en.md

* Update README_en.md

* Update README_en.md

* Update __init__.py

* Replace OMP with std::thread (#3309)

* fix bug and codestyle

* save change

* change code style

* fix conflict

* change h file

* Update tokenizer.cc

Co-authored-by: zhoushunjie <[email protected]>
Co-authored-by: Zeyu Chen <[email protected]>

* update tipc log (#3333)

* Remove unused function of Pipelines (#3330)

* update CodeGen doc (#3299)

* update doc

* update doc

* update docs

Co-authored-by: 骑马小猫 <[email protected]>

* fix tipc log (#3337)

* [MoE] Fix recompute & communication api (#3338)

* update moe recompute.

* [few-shot] fix typo and failed links (#3339)

Co-authored-by: Zhong Hui <[email protected]>

* [New Model]add t5-encoder-model (#3168)

* add t5-encoder-model

* update t5model

* update t5encoder & test modeling

* update t5

* update type hinting

* update cache type annotation

* Update retrieval based classification README.md (#3322)

* Update retrieval based classification README.md

* Revert predict.py

* Update cpu predict script

* restore gpu config

* Fix TIPC log path (#3347)

* Upgrade Neural Search README.md (#3350)

* support layoutxlm re dygraph to static (#3325)

* support layoutxlm re dygraph to static

* fix error

* upgrade-modeling-output (#3305)

* upgrade-modeling-output

* fix codestyle

* Compression API supports ELECTRA (#3324)

* supports electra

* fix typo

* [FasterGeneration] MBart supports dy2sta (#3356)

* unimo unittests (#3349)

* [Benchamrk] Fix fuse_transformer option of TIPC (#3358)

* Fix the README description of Pipelines & Neural Search (#3353)

* Fix the README description

* Update Pipelines README.md

* Update Docker README.md

* Add more details for ranking model

* supports distribute (#3361)

* Fix the semantic search example mistakes (#3363)

Co-authored-by: Zeyu Chen <[email protected]>

* [BugFix] Fix amp usage for evaluation. (#3303)

* fix eval of amp usage.

* fix

* [MoE] Fix distributed wait api (#3365)

* Fix gpt example attention mask (#3240)

* add hf ds and upgrade example

* fix attention mask

* update

* update attention mask

* fix static attention mask

* Fix erniegen no model_config_file (#3321)

* fix

* rm save_pretrained

* fix tipc log for benchmark and upate bigru_crf config (#3373)

* fix tipc log

* fix tipc log and upate bigru_crf config

* add t5 encoder model (#3376)

* MBART supports freeze multi-lingual model when dy2sta (#3367)

* fix dataloader memory overflow

* add warning

* Update README_en.md (#3375)

edit typo

Co-authored-by: Zeyu Chen <[email protected]>

* Improve CodeGen (#3371)

* Add codegen unittests (#3348)

* add codegen unittests

* fix codegen

* update

* [BugFix] fix supporting `OrderedDict` bug in paddle.jit module (#3364)

* convert keys to `__dict__`

* use fields to get keys

Co-authored-by: Guo Sheng <[email protected]>

* 【Hackathon + GradientCache】 (#1799)

* gradient_cache

* gradient_cache

* gradient_cache

* gradient_cache

* data

* train_for_gradient_cache

* add

* add

* add

* 修改

* 修改

* update

* update

* update

* update

* Update README_gradient_cache.md

* Update README_gradient_cache.md

* Update README_gradient_cache.md

* feat: modified the code

* fix: delete useless code

* feat: added requirements.txt

* feat: modify readme

* feat: modify some code

* feat: code style

* feat: add function

* feat: add licence

* feat: add comments

* Update README_gradient_cache.md

* feat: modify readme

* feat: modify readme

* fix: copyright

* fix: yapf

* feat: modify readme

* feat: modify readme

* feat: delete useless code

* feat: add new explain

Co-authored-by: 吴高升 <[email protected]>
Co-authored-by: 吴高升 <[email protected]>

* [TIPC] Add scripts for npu and xpu, test=develop (#3377)

* add scripts for xpu and npu

* add npu/xpu args

* add script for xpu

* add npu/xpu args to predict.py

* fix codestyle ci bug

* add copyright

* fix copyright_checker

* Add ERNIE-LayoutX (#3183)

* Add ernie-layoutx

* simplify code

* simplify code

* support batch input

* add word_boxes support

* Update docs

* update

* Update README.md

* Udpate README.md

* Update README.md

* Update README.md

* [Dygraph] Support sharding stage2/3+dp in GPT-3 model (#2471)

* add sharding+dp

* update

* code style check

Co-authored-by: gongenlei <[email protected]>

* complete t5 more output (#3370)

* fix gpt N4C32 dp script bug (#3392)

* codestyle

* Update README.md of neural search (#3391)

* Update artist model activateion (#3106)

* update

* rename

* fix gpt ut (#3407)

* add qg example

* delete useless scripts

* delete .sh files in t5 dir

* normalize t5 naming

* rewrite run_gen.py to train.py and predict.py in unimo-text

* Update README_cn.md (#3413)

* fix bigru crf offset index error (#3418)

* modified according to zeyang's comments

* modified according to zeyang's comments

* fix bert unittest bug (#3422)

* fix bert unittest bug

* change token_labels -> sequence_labels

* [BugFix]Fix ernie tokenizer unittest (#3423)

* fix bert unittest bug

* change token_labels -> sequence_labels

* update ernie tokenizer max_input_size

* update qg example readme

* fix pillow deperate warning (#3404)

Co-authored-by: gongenlei <[email protected]>

* Update taskflow.py (#3424)

fix typo

* fix bug of debug mode (#3417)

* rewrite unimo-text/predict.py to retrain only the prediction function

* support paddle serving http deploy for text classification (#3378)

* add_http_deploy

* [prompt] add doc (#3362)

* modified according to zeyang's comments, 20221010

* [few-shot] fix script for multi_class and fix input type for windows (#3426)

* Update README_cn.md

* adjust the position of the experiment' result

* support mlu training (#3431)

* support mlu training

* [mlu] add mlu config in rnn and ernie-1.0 README.

* remove the tcn for the paddlenlp (#3435)

* add qg-taskflow

* fix code style

* Add multi type files index update example for pipelines (#3439)

* [MLU] support SQuAD_Bert with mlu device (#3434)

* Update FAQ Finance Paddle Serving dependencies (#3430)

* Add batch prediction for pipelines (#3432)

* Add batch prediction for pipelines

* Fix some hardcode problem& Update comments

* Support past_key_values argument for Electra (#3411)

* unit test pass; fix yapf

* change docstring

Co-authored-by: 骑马小猫 <[email protected]>
Co-authored-by: Guo Sheng <[email protected]>

* modified according to zeyang's comments

* refine gpt (#3447)

* fix some typos in qg-example readme

* Fix #3446 (#3457)

* update Pillow version

* compare version

* [NEW Features] feature_extraction and processor support from_pretrained (#3453)

* update

* add import

* Update README.md and optimize DocPrompt postprocess (#3441)

* Update README.md

* optimize sort

* update

* Update

* Update

* Update

* Update

* Update

* Update

* update

* update

* Add english docs and rename ernie_layout

* Add english docs and rename ernie_layout

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* Update taskflow.md

* update

* add symbolic link for ernie_layout

* Update README.md

Co-authored-by: wj-Mcat <[email protected]>
Co-authored-by: yujun <[email protected]>
Co-authored-by: 吴高升 <[email protected]>
Co-authored-by: limingshu <[email protected]>
Co-authored-by: chenxiaozeng <[email protected]>
Co-authored-by: tianxin <[email protected]>
Co-authored-by: Guo Sheng <[email protected]>
Co-authored-by: bruce0210 <[email protected]>
Co-authored-by: wuhuachaocoding <[email protected]>
Co-authored-by: Zhong Hui <[email protected]>
Co-authored-by: wawltor <[email protected]>
Co-authored-by: liu zhengxi <[email protected]>
Co-authored-by: kztao <[email protected]>
Co-authored-by: Jack Zhou <[email protected]>
Co-authored-by: paopjian <[email protected]>
Co-authored-by: gongenlei <[email protected]>
Co-authored-by: lugimzzz <[email protected]>
Co-authored-by: Jiaqi Liu <[email protected]>
Co-authored-by: WangZhen <[email protected]>
Co-authored-by: Thomas Young <[email protected]>
Co-authored-by: Zeyu Chen <[email protected]>
Co-authored-by: zhengya01 <[email protected]>
Co-authored-by: Roc <[email protected]>
Co-authored-by: Noel <[email protected]>
Co-authored-by: zhoujun <[email protected]>
Co-authored-by: Liujie0926 <[email protected]>
Co-authored-by: westfish <[email protected]>
Co-authored-by: Septilliony <[email protected]>
Co-authored-by: Elvis Stuart <[email protected]>
Co-authored-by: 吴高升 <[email protected]>
Co-authored-by: duanyanhui <[email protected]>
Co-authored-by: Haohongxiang <[email protected]>
Co-authored-by: Yam <[email protected]>
Co-authored-by: sneaxiy <[email protected]>
Co-authored-by: alkaid <[email protected]>
Co-authored-by: Chenxiao Niu <[email protected]>
Co-authored-by: qipengh <[email protected]>
Co-authored-by: Sijun He <[email protected]>
@LemonNoel LemonNoel deleted the prompt_doc branch December 9, 2022 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ZHUI ZHUI ZHUI approved these changes

Assignees

@LemonNoel LemonNoel

Labels
prompt
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@LemonNoel @ZHUI