[Bugfix] Adds outlines performance improvement

[lynkz-matt-psaltis]

Borrows outlines upgrade from #4109 against Guide and detects state resets to clear cache

[Peng-YM]

It seems to me that instruction has no attribute named id

[maxdebayser]

Yes, with outlines==0.0.46 there doesn't seem to be and id attribute. I'm testing with

cacheKey = hash(tuple(allowed_tokens))

instead

[maxdebayser]

I'm doing some of my testing with the cpu backend. I don't know if there is a better way to test for the availability of pin_memory() but I'm running it with:

allowed_tokens_tensor = torch.from_numpy(np_allowed_tokens)
try:
    allowed_tokens_tensor = allowed_tokens_tensor.pin_memory()
except NotImplementedError:
    pass

[maxdebayser]

I'm testing this change with this request:

curl http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{
    "model": "facebook/opt-125m",
    "prompt": ["Here is an example of a JSON document representing a user record: "],
    "max_tokens": 1000,
    "min_tokens": 900,
    "temperature": 0,
    "guided_decoding_backend": "outlines",
    "response_format": {"type":"json_object"},
    "frequency_penalty": 2.0,
    "presence_penalty": 2.0
  }' 

I'm using a very small model on purpose so that a potential performance improvement can stand out more against the time it takes to generate to run the model forward().

Excluding the first call and averaging the next 3 calls the result is 22s with the baseline and the changes in this PR, so I'm not seeing a big difference. But perhaps this is a case that doesn't benefit from the caching in this PR. FYI, this test was run on a 80GB A100.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lynkz-matt-psaltis.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hmellor]

Closing as this PR is quite old and conflicts with main. If you want to continue working in this PR (or open a new one) please feel free to do so!