lecture10-Transformer解读 #50
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ZhikangNiu
requested changes
Feb 1, 2023
There was a problem hiding this comment.
- 序号问题
- 图片名称全部改为英文
- 可以附上参考资料
- Transformer国内外有很多写的比较好的blog可以参考解读下，可以更深层次解读下
- MultiheadAttention 在pytorch提供了该方法，可以简单介绍下 @nowadays0421
source/第十章/Transformer 解读.md
Outdated
|
|
||
| 针对上述两个问题，2017年，Vaswani 等人发表了论文《Attention Is All You Need》，抛弃了传统的 CNN、RNN 架构，提出了一种全新的完全基于 attention 机制的模型——Transformer，解决了上述问题，在较小的时间成本下取得了多个任务的 the-state-of-art 效果，并为自然语言处理任务提供了新的思路。自此，attention 机制进入自然语言处理任务的主流架构，众多性能卓越的预训练模型都基于 Transformer 架构提出，例如 BERT、OpenAI GPT 等。 | ||
|
|
||
| 本文将从模型原理及代码实现上讲解该模型，并着重介绍代码实现。需要注意的是，由于 Transformer 源代码使用 TensorFlow 搭建，此处选择了哈佛大学 harvardnlp 团队基于 Pytorch 框架开发的 Annotated Transformer 代码进行讲解，以帮助大家了解 Transformer 的实现细节。 |
There was a problem hiding this comment.
这里可不可以把Annotated Transformer的链接也附上，做成超链接
source/第十章/Transformer 解读.md
Outdated
Comment on lines
21
to
23
| ① RNN 为单向依序计算，序列需要依次输入、串行计算，限制了计算机的并行计算能力，导致时间成本过高。 | ||
|
|
||
| ② RNN 难以捕捉长期依赖问题，即对于极长序列，RNN 难以捕捉远距离输入之间的关系。虽然 LSTM 通过门机制对此进行了一定优化，但 RNN 对长期依赖问题的捕捉能力依旧是不如人意的。 |
source/第十章/Transformer 解读.md
Outdated
|
|
||
| Transformer 是针对自然语言处理的 Seq2Seq（序列到序列）任务开发的，整体上沿用了 Seq2Seq 模型的 Encoder-Decoder（编码器-解码器）结构，整体架构如下： | ||
|
|
||
| <img src="./figures/transformer_整体架构.png" alt="image-20230127193646262" style="zoom:50%;" /> |
There was a problem hiding this comment.
- 图片名称使用全英文，带中文会导致后面在线网页导出有问题
- 可以加上align="center"，使得图片居中
source/第十章/Transformer 解读.md
Outdated
|
|
||
| ## Mask | ||
|
|
||
| Transformer 是一个自回归模型，类似于语言模型，其将利用历史信息依序对输出进行预测。例如，如果语料的句对为：① BOS 我爱你 EOS；② BOS I like you EOS。则 Encoder 获取的输入将会是句①整体，并输出句①的编码信息，但 Decoder 的输入并不一开始就是句②整体，而是先输入起始符 BOS，Decoder 根据 BOS 与 Encoder 的输出预测 I，再输入 BOS I，Decoder 根据输入和 Encoder 的输出预测 like。因此，自回归模型需要对输入进行 mask（遮蔽），以保证模型不会使用未来信息预测当下。关于自回归模型与自编码模型的细节，感兴趣的读者可以下来查阅更多资料。 |
There was a problem hiding this comment.
- 序号问题
- BOS这种作为特殊字符，是不是应该以加上括号区分开来
- 关于关于自回归模型与自编码模型的细节，感兴趣的读者可以下来查阅更多资料，可以给出相关链接
ZhikangNiu
approved these changes
Feb 9, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
上传了第十章常见代码解读的Transformer解读部分。