Cubic sampling w/ curve param

[kalomaze]

This adds upon the original Quadratic Sampling method with an additional parameter that I've labeled "smoothing_curve".

The idea is to enable even lower smoothing_factor values than ~0.25ish to work well; we do this by applying a cubic transformation to compensate, which seems to make the falloff steeper.

Not ready to merge yet, needs empirical testing from users. My hope is that you can fully avoid having to use truncation schemes and go for a fully "smooth" transformation to the distribution across different models.

The higher the smoothing_curve, the steeper the fall off (so it becomes harsher).

2024-02-19_15-42-06.mp4

Brief tests on a 7b are showing that it does in fact help to make lower smoothing_factor values coherent in practice (so far at least).

1.0 smoothing_curve is the "old" behavior and has no effect.

[BadisG]

When you go for 3 or higher on the smoothing_curve, it doesn't work anymore

[kalomaze]

When you go for 3 or higher on the smoothing_curve, it doesn't work anymore

I can't reproduce this issue, through llama.cpp_HF at least.

[Myobu1]

When you go for 3 or higher on the smoothing_curve, it doesn't work anymore

I can't reproduce this issue, through llama.cpp_HF at least.

I have the exact same issue on Ooba. Setting smoothing curve to lower than 3, it works fine. The moment I go at or above it, I start getting errors regarding RuntimeError: probability tensor contains either inf, nan or element < 0 and inability to output replies after 5 attempts on Silly.

Edit: Apparently if you have any banned tokens the issue will occur.

[Ph0rk0z]

I get nan error as well. using exl2_HF. utils.py line 2734

Its a problem from transformers.

next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)

2.99 is the highest I can go before this error. But so far I was able to get the factor down to .05 with that value.

[Ph0rk0z]

I only got 2 braincells to knock together, but to me

k = (3 - self.smoothing_curve) / 2

3-3 is 0 and then thats 0/2 which makes NaN

Btw: Changing it to 10 did fix it for me, up to 9.99 ofc. I am getting best results with .20-.2X and 1.04 because all I can do is watch the PR video go by token distribution pictured. Also .02 and 4.82-5.6 was another decent point. Otherwise this removes too many tokens and gets very deterministic.

[oobabooga]

The nan error was caused by making operations with -inf^3. I solved it by keeping scores with -inf value unchanged.

[Ph0rk0z]

Will retest.

works but back to square one on how to set it.