Use less problematic whitespace token

[lapp0]

Fixes #839 #908 #690 #450

Problem

A major problem, especially with smaller language models, is the repetition problem.

For example, let's say a model is generating json and must provide 12 space tokens for indentation in json output. Often a language model will assign a high probability to a 13th space token, and do the same for a 14th space, and then enter an infinite space generation loop.

This is a problem with NLG that has been known for half a decade, but only has mitigations (mirostat, repetition penalty, using hundreds of billions of weights, etc), no absolute solutions (except for structured generation)

Solution

For structured json generation, we set a sane default whitespace pattern of r"[ ]?". This removes all newlines and indentation. It disallows any syntactic whitespace beyond a single space separator.

Users can still set the argument whitespace_pattern= if they want different behavior

[rlouf]

Great, thank you!

[timothylimyl]

Thanks for making the PR!