Skip to content

Define non-positive temperature behavior#720

Merged
prusnak merged 1 commit into
ggml-org:masterfrom
ivanstepanovftw:non-positive-temperature
Apr 3, 2023
Merged

Define non-positive temperature behavior#720
prusnak merged 1 commit into
ggml-org:masterfrom
ivanstepanovftw:non-positive-temperature

Conversation

@ivanstepanovftw

Copy link
Copy Markdown
Contributor

Fixes #684.

@ivanstepanovftw

Copy link
Copy Markdown
Contributor Author

Original LLaMA's generate defines temperature like so:

        if temperature > 0:
            probs = torch.softmax(logits / temperature, dim=-1)
            next_token = sample(probs, top_p=top_p, top_k=top_k)
        else:
            next_token = torch.argmax(logits, dim=-1)

@prusnak

prusnak commented Apr 2, 2023

Copy link
Copy Markdown
Contributor

@abetlen can you please also test this one?

@abetlen

abetlen commented Apr 2, 2023

Copy link
Copy Markdown
Collaborator

@prusnak tested and can confirm this works as well.

@prusnak prusnak merged commit cd7fa95 into ggml-org:master Apr 3, 2023

@Fabio3rs Fabio3rs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This alternative seems to me to harmonize better with the rest of the code

@ivanstepanovftw ivanstepanovftw deleted the non-positive-temperature branch May 5, 2023 17:43
Qeeweew pushed a commit to Qeeweew/llama.cpp that referenced this pull request May 17, 2024
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setting temp=0 does not work as expected

4 participants