Text Pattern Prompts

[realdavidvega]

This PR brings the feature of creating more deterministic OpenAI text completions based on a prompt String and a Regex. Right now only available for the Kotlin/JVM target. It's inspired on Matt Rickard's ReLLM python library.

It works iterating to create the final completion, by creating a logitBias map by filtering all possible tokens that match partially the Regex and sending to OpenAI the original prompt plus a partial completion for every step. Each partial completion has a maximum size of maxTokens=1.

Each partially matching token in logitBias gets a value of 100, telling the model that it's one of the exclusive tokens to choose in the completion. Take into account, that by the day of this PR, logitBias is limited by OpenAI to 300 tokens, therefore we can only send at most that size.

In the function there's also an optional limit for the maximum number of tokens generated, with a default value of maxNewTokens = 30, and a flag in case we want to stop at the full match of the Regex, with default value of stopAfterMatch = true.

Example of usage:

ai {
    val goal = "Return the first three letters of the alphabet in a json array: "
    val response: String = patternPrompt(
        prompt = goal,
        pattern = Regex("""\["[a-z]", "[a-z]", "[a-z]"]"""),
        maxNewTokens = 20
    )
    val list: List<String> = Json.decodeFromString(response)
    println(list) //  ["a", "b", "c"]
}.getOrElse { println(it) }

Be mindful that for very complex prompts or regexes, this can result on heavy OpenAI usages.

It would be interesting to explore this approach with more complex ways of matching those tokens (like grammars), working with local models with no logitBias size or API limitations, or incorporating other logitBias strategies to other functions of the DSL.

cc/ @xebia-functional/team-ai

[raulraja]

This looks great @realdavidvega! Can we use this to solve the current problem we have with the JSON responses?

[realdavidvega]

This looks great @realdavidvega! Can we use this to solve the current problem we have with the JSON responses?

It could be useful to some use cases, but maybe not all of them. If the prompt with context is too large, the cost would be too high. Also, the limitation of 300 tokens in OpenAI's logit_bias is giving us less options to control the completion.

[raulraja]

thanks @realdavidvega !