-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support chat modality in attempts #644
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
todo:
|
todo:
|
…ast test to be clearly a 5x5 test
…e handling constructor vars
…nversation has started
…g and float-returning
jmartin-tech
approved these changes
Jul 1, 2024
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
new pattern: chat histories are supported and managed entirely in
Attempt
Expected use
* an attempt tracks a seed prompt and responses to it
* there's a 1:1 relationship between attempts and source prompts
* attempts track all generations
* this means messages tracks many histories, one per generation
* for compatibility, setting Attempt.prompt will set just one turn, and this is unpacked later
when output is set; we don't know the # generations to expect until some output arrives
* to keep alignment, generators need to return aligned lists of length == #generations
Implementation is by selectively overriding access to
Attempt.prompt
andAttempt.outputs
. The hope is to retain existing compatibility across garak, while also tracking message histories for plugins that use them.Managing histories means output tracking, so we need a 1:1 mapping from prompt to output. Because there are typically multiple output generations to one seed prompt, this needs to managed. Two options:
Attempt
tracks many histories, one for each of the number of output generations requested. This requires being aware of / sensitive to the number of generations.Attempts
, one per generationBecause 2. breaks the underlying data model, we prefer 1. This does mean being aware of the number of outputs, so a few consistency checks are built in to
attempt
, as well as silently upgrading from one to multiple histories once we know the number of outputs. Tests are included.Patterns/expectations for Attempt getting:
.prompt
- returns the first user prompt.outputs
- returns the most recent model outputs.latest_prompts
- returns a list of the latest user prompts.all_outputs
- returns all model outputs from all dialogue turnsPatterns/expectations for Attempt setting:
.prompt
- sets the first prompt, or fails if this has already been done.outputs
- sets a new layer of model responses. silently handles expansion of prompt to multiple histories.latest_prompts
- adds a new set of user promptspending work:
atkgen
has a roll-your-own chat model, which is not compatibleresolves #109