Adding Locally Typical Sampling (i.e. typical_p in transformers and TGI)

[seongminp]

Hello. First of all, thank you for your wonderful work.

I was curious if adding locally typical sampling was on the roadmap.
If not, would you be interested in a PR adding the functionality?
I think adding typical_p will be trivial, but wanted to ask first before working on it.

Personally, I'm having great success with typical_p in transformers and text-generation-inference.
I want to switch to vLLM but I can't go back to top_p.

[tjohnson31415]

I am interested in support for typical_p sampling as well.

With the changes that added logits_processors as an extension point for sampling in #1469, it is possible to use Transformer's TypicalLogitsWarper with a little wrapper:

from transformers.generation.logits_process import TypicalLogitsWarper

class TypicalLogitsWarperWrapper:
    def __init__(self, mass: float):
        self.warper = TypicalLogitsWarper(mass=mass)
        
    def __call__(self, token_ids: List[int], logits: torch.tensor) -> torch.tensor:
        # transformers warpers assume tensors of shape (batch_size, vocab_size)
        # and the typical warper doesn't use input_ids
        return self.warper(input_ids=None, scores=logits.reshape((1, -1)))

sampling_params = SamplingParams(
    logits_processors=[TypicalLogitsWarperWrapper(mass=0.7)],
)