Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[asr] optimize attention cache used for attention ; 0-dim tensor for model export #2124

Merged
merged 3 commits into from
Jul 12, 2022
Merged

[asr] optimize attention cache used for attention ; 0-dim tensor for model export #2124

merged 3 commits into from
Jul 12, 2022

Conversation

zh794390558
Copy link
Collaborator

@zh794390558 zh794390558 commented Jul 8, 2022 •
edited
Loading

PR types

Performance optimization

PR changes

Models

Describe

  1. 优化后相比优化前相对提升：9.9%
conformer_wenetspeech - python attention_rescoring 0.5 16 -1 0.8712686765174202(utts=40) -
conformer_wenetspeech（reduce attention cache） - python attention_rescoring 0.5 16 -1 0.7847489114800089(utts=40)  
  1. using 0-dim tensor (tensor(0,0,0,0) ) to simple to_static conformer model export
  2. fix cli to support scp batch process
  3. add pybind11 to install deps.

@zh794390558 zh794390558 added this to the r1.2.0 milestone Jul 8, 2022
@zh794390558 zh794390558 changed the title [asr] optimize attention cache used for attention [asr] optimize attention cache used for attention ; 0-dim tensor for model export Jul 8, 2022
Jackwaterveg
Jackwaterveg reviewed Jul 11, 2022
View reviewed changes
paddlespeech/s2t/modules/encoder.py
xs = paddle.cat((subsampling_cache, xs), dim=1)
else:
cache_size = 0
elayers = paddle.shape(att_cache)[0]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

subsampling_cache 不用了？

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

缓存的计算量确实不大，不过可能也可以节省一定时间

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

合并到att_cache里了，会降低一些计算。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok，理解了

Jackwaterveg
Jackwaterveg approved these changes Jul 11, 2022
View reviewed changes
Copy link
Contributor

@Jackwaterveg Jackwaterveg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zh794390558 zh794390558 marked this pull request as ready for review July 12, 2022 08:30
@zh794390558 zh794390558 modified the milestones: r1.2.0, r1.1.0 Jul 12, 2022
Jackwaterveg
Jackwaterveg approved these changes Jul 12, 2022
View reviewed changes
Copy link
Contributor

@Jackwaterveg Jackwaterveg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zh794390558 zh794390558 merged commit e62cbc4 into PaddlePaddle:develop Jul 12, 2022
@zh794390558 zh794390558 deleted the new_api branch July 12, 2022 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jackwaterveg Jackwaterveg Jackwaterveg approved these changes

Assignees
No one assigned
Labels
CLI Demo Installation S2T asr/st Server
Projects
None yet
Milestone
r1.1.0
Development

Successfully merging this pull request may close these issues.
2 participants
@zh794390558 @Jackwaterveg