Update models.transformers to use SequenceGeneratorAdapter and OutlinesLogitsProcessors
#966
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lapp0
changed the title
outlines.models.transformers to use SequenceGeneratorAdapter and OutlinesLogitsProcessors
lapp0
commented
Jun 13, 2024
lapp0
commented
Jun 13, 2024
| @@ -39,8 +40,9 @@ def regex(model, regex_str: str, sampler: Sampler = multinomial()): | |||
|
|
|||
|
|
|||
| @regex.register(MLXLM) | |||
| def regex_mlxlm( | |||
| model: MLXLM, | |||
| @regex.register(Transformers) | |||
There was a problem hiding this comment.
the _unified dispatchers will become the default dispatcher as a next step. In this PR it's just used by MLXLM and Transformers
lapp0
force-pushed
the
transformers-use-logits-processor
branch
18 times, most recently
from
d94527a
to
5bd8832
Compare
lapp0
force-pushed
the
transformers-use-logits-processor
branch
from
5bd8832
to
1537695
Compare
lapp0
force-pushed
the
transformers-use-logits-processor
branch
4 times, most recently
from
d0fb1a6
to
0167700
Compare
lapp0
force-pushed
the
transformers-use-logits-processor
branch
2 times, most recently
from
8f9c317
to
6ea583e
Compare
rlouf
reviewed
Jun 19, 2024
| @@ -30,4 +30,55 @@ tokenizer = AutoTokenizer.from_pretrained("gpt2") | |||
| model = models.Transformers(llm, tokenizer) | |||
| ``` | |||
|
|
|||
| # Using Logits Processors | |||
There was a problem hiding this comment.
We'll need to improve the documentation to reach something similar to the lamacpp integration's.
There was a problem hiding this comment.
We should plan a restructuring and cleaning up of documentation in a separate issue. I could share some ideas in call on how we might approach this.
In this case, a lot of information documented for llamacpp applies to all other models including transformers. We shouldn't repeat ourselves. We should explain the behavior of all models generally, highlight the models differences with a feature table, and document only transformers specific information on its documentation page.
There was a problem hiding this comment.
What I meant was listing the main arguments to you can pass when initialising and calling the model, cf https://outlines-dev.github.io/outlines/reference/models/llamacpp/
lapp0
commented
Jun 20, 2024
lapp0
force-pushed
the
transformers-use-logits-processor
branch
2 times, most recently
from
f5ae15e
to
b75beeb
Compare
This was referenced Jun 21, 2024
This was referenced Jun 23, 2024
rlouf
pushed a commit
that referenced
this pull request
Jun 30, 2024
….py (#998) A lot of these fixes were intended for #966 however that's blocked until there's a new `transformers` release. These improvements are general to all models and will enable PRs resolving #806 and #965 # Structure of `OutlinesLogitsProcessor` The goal is to create a base class which allows a logits processors to be implemented once and used for any `outlines.models` inference library. To accomplish this we must normalize the input array. It must have a consistent type (`torch.Tensor`) and consistent dimensionality (2). We can normalize both of these simply, and without any copy operations. `mlx.core.array`, `numpy.array`, and `torch.Tensor` all support [pythons array standard `__dlpack__`](https://data-apis.org/array-api/latest/API_specification/generated/array_api.array.__dlpack__.html). This standard allows for casting between array types without copying. `torch.Tensor` is the only input type which cannot always be cast to any other type because torch tensors may live in GPU memory. Therefore, we cast all arrays to `torch.Tensor`, implement logits processors using torch methods, and convert back to the original array type in `OutlinesLogitsProcessor`. See docstring of `OutlinesLogitsProcessor.__call__()` for more details. # Detailed Changes - Rename `BaseLogitsProcessor` to `OutlinesLogitsProcessor` - Ensure `OutlinesLogitsProcessor.process_logits()` is always passed a 2D batch request with `torch.Tensor` logits and `List` input_ids. Also clean up code to be more readable in `OutlinesLogitsProcessor__call__()` - Ensure `FSMLogitsProcessor` allows unstable sequence ordering (beam search in transformers and vLLM change the order of sequences) - Update `tests/generate/test_generate.py` to cover more permutations of - regex / text - batch / single - greedy / multinomial / beam search - `stream()` / `generate()` - Ensure performance stability with difference array libraries through `benchmark_processors.py`
lapp0
force-pushed
the
transformers-use-logits-processor
branch
from
b75beeb
to
cdc78f0
Compare
lapp0
commented
Jun 30, 2024
lapp0
commented
Jun 30, 2024
lapp0
force-pushed
the
transformers-use-logits-processor
branch
4 times, most recently
from
b495c2c
to
326ab77
Compare
lapp0
force-pushed
the
transformers-use-logits-processor
branch
2 times, most recently
from
c24b1fa
to
32319df
Compare
lapp0
force-pushed
the
transformers-use-logits-processor
branch
from
32319df
to
7d43bbd
Compare
rlouf
approved these changes
Jul 15, 2024
|
Very good work, thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In draft until huggingface/transformers#31448 makes it into a newtransformersreleaseFixes #1021
Fixes #789
Fixes #806 (does everything except remove the issues requirement of "remove
torch")Closes #910 (device inconsistency issue handled through other means)
Problem
SequenceGeneratorifinput_idsis empty, it fails.models.transformershas an inconsistent implementation with other modules making the codebase harder to manage.Solution
SequenceGeneratorAdapterfor transformers instead ofSequenceGeneratorTransformers.generateandTransformers.streamwhich usemodel.generate(logits_processor=...)argument withoutlines.processors.OutlinesLogitsProcessorAdditional Changes
Temporarily disablesstop_atargument for transformers and implements test to determine whether upstream is fixed (stop_stringsArgument inmodel.generate()Results in Exception if Generation Completes Withoutstop_stringBeing Generated huggingface/transformers#31435)seedinstead ofrngoutlines.generate.cfgfor now (Using context-free grammars to guide generation does not work #959)TODO:
GenerationConfigdefaultmax_tokensbeing 20rngwithseedin all tests and all documentation.transformersfix forstop_stringsArgument inmodel.generate()Results in Exception if Generation Completes Withoutstop_stringBeing Generated huggingface/transformers#31435 sostop_atworks.