【Hackathon 7th No.43】TokenizerFast for Qwen2

[yinfan98]

PR types

New features

PR changes

Others

Description

添加qwen2 tokenizer fast

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 89.18919% with 4 lines in your changes missing coverage. Please review.

Project coverage is 52.85%. Comparing base (d455181) to head (92e7e24).
Report is 13 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/qwen2/tokenizer_fast.py	90.00%	2 Missing ⚠️
paddlenlp/transformers/tokenizer_utils_base.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9532      +/-   ##
===========================================
- Coverage    53.19%   52.85%   -0.35%     
===========================================
  Files          700      700              
  Lines       110757   110412     -345     
===========================================
- Hits         58921    58358     -563     
- Misses       51836    52054     +218     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.