Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sequence post processor #771

Merged
merged 6 commits into from
May 23, 2024
Merged

Add sequence post processor #771

merged 6 commits into from
May 23, 2024

Commits on May 23, 2024

  1. Add Sequence PostProcessor 

    Required by https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/
    @xenova
    xenova committed May 23, 2024
    Configuration menu
    Copy the full SHA
    c1332f8 View commit details
    Browse the repository at this point in the history

  2. Support return_token_type_ids

    @xenova
    xenova committed May 23, 2024
    Configuration menu
    Copy the full SHA
    c018f27 View commit details
    Browse the repository at this point in the history

  3. Add llama3 tokenizer to unit tests

    @xenova
    xenova committed May 23, 2024
    Configuration menu
    Copy the full SHA
    d68afe2 View commit details
    Browse the repository at this point in the history

  4. Add test for allowing user to request for token type ids

    @xenova
    xenova committed May 23, 2024
    Configuration menu
    Copy the full SHA
    0e5fb41 View commit details
    Browse the repository at this point in the history

  5. Add JSDoc

    @xenova
    xenova committed May 23, 2024
    Configuration menu
    Copy the full SHA
    3b2f5cd View commit details
    Browse the repository at this point in the history

  6. Update generate_tests.py

    @xenova
    xenova committed May 23, 2024
    Configuration menu
    Copy the full SHA
    078672b View commit details
    Browse the repository at this point in the history