Skip to content

Rename additive_repetition_penalty to presence_penalty, add frequency_penalty#4376

Merged
oobabooga merged 12 commits intooobabooga:mainfrom
tdrussell:repetition_penalty
Oct 25, 2023
Merged

Rename additive_repetition_penalty to presence_penalty, add frequency_penalty#4376
oobabooga merged 12 commits intooobabooga:mainfrom
tdrussell:repetition_penalty

Conversation

@tdrussell
Copy link
Contributor

@tdrussell tdrussell commented Oct 24, 2023

See the discussion in #3627.

These repetition penalty settings work the same as the ones in the OpenAI chat completion API.

Checklist:

@tdrussell
Rename additive_repetition_penalty to presence_penalty
30b19b3 
This matches the OpenAI API name for this setting.
@tdrussell
Change presence_penalty slider range.
932110b 
Make it match the OpenAI setting.
@tdrussell
Add frequency_penalty param everywhere presence_penalty exists
cc469e6
@tdrussell
Implement frequency_penalty setting
27a6dde
@tdrussell
Update API generate_params
95ad00b
@tdrussell
Fix dimensions issue with torch.unique()
fa6adcb
@tdrussell
Update slider ranges for presence and frequency penalties
b34a066 
First, make them >=0 to match repetition_penalty. Second, raise the
maximum presence_penalty to 4. In some limited testing, I saw that
presence penalty needed to be slightly over 3 to match the probability
shift of a standard repetition_penalty of 1.2. So a max of 2 wasn't
enough.
@kingbri1
Copy link
Contributor

kingbri1 commented Oct 25, 2023 •
edited
Loading

Spelling error in 95ad00b

frquency_penalty should be frequency_penalty

@tdrussell
Copy link
Contributor Author

tdrussell commented Oct 25, 2023

Thanks for noticing, I fixed it.

@oobabooga
Copy link
Owner

oobabooga commented Oct 25, 2023

Thanks a lot for adding this parameter, @tdrussell. I took the opportunity to also connect the frequency and presence penalty sliders to the llama.cpp loader, which implements them independently in its C++ backend.

I also made a sanity check by comparing the logits in the main branch and this PR with additive_repetition_penalty / presence_penalty set to 0.5, and the logits turned out identical. It's nice that you also handled the batch_size > 1 case.

@oobabooga oobabooga merged commit 72f6fc6 into oobabooga:main Oct 25, 2023
@DocShotgun
Copy link

DocShotgun commented Oct 30, 2023

Hmm, I'm wondering if there's anything different about the way OAI implements frequency_penalty or if our models are just inherently less resilient, but after trying a few different L2 models of varying sizes (13b and 70b), it seems like things go off their rocker pretty fast with frequency_penalty set to anything above a tiny value. Even with numbers as low as 0.2 I was starting to get some strangeness in the outputs.

Also mirostat + frequency_penalty really doesn't mesh well. Goes on an insane rant with frequency_penalty set at 0.1 lol.

Was wondering if there were any suggested values/ranges for these settings?

@kingbri1 kingbri1 mentioned this pull request Nov 1, 2023
Open
1 task
Vechtomov pushed a commit to Vechtomov/text-generation-webui that referenced this pull request Nov 9, 2023
@Begelit
Squashed commit of the following:
2273473 
commit e18a046
Author: kabachuha <artemkhrapov2001@yandex.ru>
Date:   Sat Nov 4 22:12:51 2023 +0300

    fix openai extension not working because of absent new defaults (oobabooga#4477)

commit b7a409e
Merge: b5c5304 fb3bd02
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Sat Nov 4 15:04:43 2023 -0300

    Merge pull request oobabooga#4476 from oobabooga/dev

    Merge dev branch

commit fb3bd02
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Sat Nov 4 11:02:24 2023 -0700

    Update docs

commit 1d8c7c1
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Sat Nov 4 11:01:15 2023 -0700

    Update docs

commit b5c5304
Merge: 262f8ae 40f7f37
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Sat Nov 4 14:19:55 2023 -0300

    Merge pull request oobabooga#4475 from oobabooga/dev

    Merge dev branch

commit 40f7f37
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Sat Nov 4 10:12:06 2023 -0700

    Update requirements

commit 2081f43
Author: Orang <51061118+Soefati@users.noreply.github.com>
Date:   Sun Nov 5 00:00:24 2023 +0700

    Bump transformers to 4.35.* (oobabooga#4474)

commit 4766a57
Author: feng lui <3090641@qq.com>
Date:   Sun Nov 5 00:59:33 2023 +0800

    transformers: add use_flash_attention_2 option (oobabooga#4373)

commit add3593
Author: wouter van der plas <2423856+wvanderp@users.noreply.github.com>
Date:   Sat Nov 4 17:41:42 2023 +0100

    fixed two links in the ui (oobabooga#4452)

commit cfbd108
Author: Casper <casperbh.96@gmail.com>
Date:   Sat Nov 4 17:09:41 2023 +0100

    Bump AWQ to 0.1.6 (oobabooga#4470)

commit aa5d671
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Sat Nov 4 13:09:07 2023 -0300

    Add temperature_last parameter (oobabooga#4472)

commit 1ab8700
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Fri Nov 3 17:38:19 2023 -0700

    Change frequency/presence penalty ranges

commit 45fcb60
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Fri Nov 3 11:29:31 2023 -0700

    Make truncation_length_max apply to max_seq_len/n_ctx

commit 7f9c1cb
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Fri Nov 3 08:25:22 2023 -0700

    Change min_p default to 0.0

commit 4537853
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Fri Nov 3 08:13:50 2023 -0700

    Change min_p default to 1.0

commit 367e5e6
Author: kalomaze <66376113+kalomaze@users.noreply.github.com>
Date:   Thu Nov 2 14:32:51 2023 -0500

    Implement Min P as a sampler option in HF loaders (oobabooga#4449)

commit fcb7017
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Thu Nov 2 12:24:09 2023 -0700

    Remove a checkbox

commit fdcaa95
Author: Julien Chaumond <julien@huggingface.co>
Date:   Thu Nov 2 20:20:54 2023 +0100

    transformers: Add a flag to force load from safetensors (oobabooga#4450)

commit c065547
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Thu Nov 2 11:23:04 2023 -0700

    Add cache_8bit option

commit 42f8163
Merge: 77abd9b a56ef2a
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Thu Nov 2 11:09:26 2023 -0700

    Merge remote-tracking branch 'refs/remotes/origin/dev' into dev

commit 77abd9b
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Thu Nov 2 08:19:42 2023 -0700

    Add no_flash_attn option

commit a56ef2a
Author: Julien Chaumond <julien@huggingface.co>
Date:   Thu Nov 2 18:07:08 2023 +0100

    make torch.load a bit safer (oobabooga#4448)

commit deba039
Author: deevis <darren.hicks@gmail.com>
Date:   Tue Oct 31 22:51:00 2023 -0600

    (fix): OpenOrca-Platypus2 models should use correct instruction_template and custom_stopping_strings (oobabooga#4435)

commit aaf726d
Author: Mehran Ziadloo <mehranziadloo@gmail.com>
Date:   Tue Oct 31 21:29:57 2023 -0700

    Updating the shared settings object when loading a model (oobabooga#4425)

commit 9bd0724
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Tue Oct 31 20:57:56 2023 -0700

    Change frequency/presence penalty ranges

commit 6b7fa45
Author: Orang <51061118+Soefati@users.noreply.github.com>
Date:   Wed Nov 1 05:12:14 2023 +0700

    Update exllamav2 version (oobabooga#4417)

commit 41e159e
Author: Casper <casperbh.96@gmail.com>
Date:   Tue Oct 31 23:11:22 2023 +0100

    Bump AutoAWQ to v0.1.5 (oobabooga#4410)

commit 0707ed7
Author: Meheret <101792782+senadev42@users.noreply.github.com>
Date:   Wed Nov 1 01:09:05 2023 +0300

    updated wiki link (oobabooga#4415)

commit 262f8ae
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Fri Oct 27 06:49:14 2023 -0700

    Use default gr.Dataframe for evaluation table

commit f481ce3
Author: James Braza <jamesbraza@gmail.com>
Date:   Thu Oct 26 21:02:28 2023 -0700

    Adding `platform_system` to `autoawq` (oobabooga#4390)

commit af98587
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Oct 27 00:46:16 2023 -0300

    Update accelerate requirement from ==0.23.* to ==0.24.* (oobabooga#4400)

commit 839a87b
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Thu Oct 26 20:26:25 2023 -0700

    Fix is_ccl_available & is_xpu_available imports

commit 778a010
Author: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>
Date:   Fri Oct 27 08:09:51 2023 +0530

    Intel Gpu support initialization (oobabooga#4340)

commit 317e2c8
Author: GuizzyQC <86683381+GuizzyQC@users.noreply.github.com>
Date:   Thu Oct 26 22:03:21 2023 -0400

    sd_api_pictures: fix Gradio warning message regarding custom value (oobabooga#4391)

commit 92b2f57
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Thu Oct 26 18:57:32 2023 -0700

    Minor metadata bug fix (second attempt)

commit 2d97897
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Wed Oct 25 11:21:18 2023 -0700

    Don't install flash-attention on windows + cuda 11

commit 0ced78f
Author: LightningDragon <lightningdragon96@gmail.com>
Date:   Wed Oct 25 09:15:34 2023 -0600

    Replace hashlib.sha256 with hashlib.file_digest so we don't need to load entire files into ram before hashing them. (oobabooga#4383)

commit 72f6fc6
Author: tdrussell <6509934+tdrussell@users.noreply.github.com>
Date:   Wed Oct 25 10:10:28 2023 -0500

    Rename additive_repetition_penalty to presence_penalty, add frequency_penalty (oobabooga#4376)

commit ef1489c
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 20:45:43 2023 -0700

    Remove unused parameter in AutoAWQ

commit 1edf321
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 13:09:03 2023 -0700

    Lint

commit 280ae72
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 13:07:17 2023 -0700

    Organize

commit 49e5eec
Merge: 82c11be 4bc4113
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 12:54:05 2023 -0700

    Merge remote-tracking branch 'refs/remotes/origin/main'

commit 82c11be
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 12:49:07 2023 -0700

    Update 04 - Model Tab.md

commit 306d764
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 12:46:24 2023 -0700

    Minor metadata bug fix

commit 4bc4113
Author: adrianfiedler <adrian_fiedler@msn.com>
Date:   Mon Oct 23 19:09:57 2023 +0200

    Fix broken links (oobabooga#4367)

    ---------

    Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>

commit 92691ee
Author: oobabooga <112222186+oobabooga@users.noreply.github.com>
Date:   Mon Oct 23 09:57:44 2023 -0700

    Disable trust_remote_code by default
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tdrussell @kingbri1 @oobabooga @DocShotgun