CU-865cn6kqn Model capabilities and AIScope simplification.

[raulraja]

This PR aims to enhance the flexibility and robustness of the ai dsl by ensuring its compatibility with local models, alongside the existing OpenAI models.

The AIScope cares about embeddings to index context in the store used in the operators. Still, it does not care about the models themselves or what they do because each model operation is independent of the scope and only uses the scope to retrieve the context in those operations where it's needed.

The changes involve a major refactoring of AIScope, gets rid of the client dependencies, moving several responsibilities to the individual model implementation, which are now accurately typed based on their capabilities. This ensures that the usage of each model is transparent and explicit in the code, eliminating any hardcoded dependencies on OpenAI models.
You can still achieve the same short syntax as before but must import it from the openai package defaults.
You can also get a hold of default models and build your own without needing a client abstraction.

Three layers of syntax are now available, including generic functions like prompt and promptMessage in AIScope that take a model as an argument or use it as the receiver, and similar functions available directly on the model and vendor defaults such as OpenAI. The syntax requests the VectorStore context or not as needed.

Additionally, I added a way to have local embeddings computed in memory without calls to openAI and added it by default used in the GPT4ALL impl which now can do all Xef can do without Open AI except function serialization.

We use the AI DJL library to load a Tokenizer from huggingface and one of their embedding models. This is, for now, JVM only and lives in the GPT4All module, but it can be generalized if we find it works and we want to stop using the paid embeddings.

Examples

New manual style without the AI block

We pass manually what otherwise is provided by the scope in an ai block:

package com.xebia.functional.xef.auto.manual

import com.xebia.functional.gpt4all.HuggingFaceLocalEmbeddings
import com.xebia.functional.xef.auto.llm.openai.OpenAI
import com.xebia.functional.xef.pdf.pdf
import com.xebia.functional.xef.vectorstores.LocalVectorStore

suspend fun main() {
  val chat = OpenAI.DEFAULT_CHAT
  val huggingFaceEmbeddings = HuggingFaceLocalEmbeddings.DEFAULT
  val vectorStore = LocalVectorStore(huggingFaceEmbeddings)
  val results = pdf("https://www.europarl.europa.eu/RegData/etudes/STUD/2023/740063/IPOL_STU(2023)740063_EN.pdf")
  vectorStore.addTexts(results)
  val result: List<String> = chat.promptMessage("What is the content about?", vectorStore)
  println(result)
}

Explicit OpenAI imports:

package com.xebia.functional.xef.auto

import com.xebia.functional.xef.auto.llm.openai.getOrElse
import com.xebia.functional.xef.auto.llm.openai.prompt
import kotlinx.serialization.Serializable

@Serializable
data class ASCIIArt(val art: String)

suspend fun main() {
    val art: AI<ASCIIArt> = ai {
        prompt( "ASCII art of a cat dancing")
    }
    println(art.getOrElse { ASCIIArt("¯\\_(ツ)_/¯" + "\n" + it.message) })
}

Mixing local text models with image models in the same ai scope:

package com.xebia.functional.xef.auto.gpt4all

import com.xebia.functional.gpt4all.GPT4All
import com.xebia.functional.gpt4all.LLModel
import com.xebia.functional.gpt4all.getOrThrow
import com.xebia.functional.gpt4all.huggingFaceUrl
import com.xebia.functional.xef.auto.PromptConfiguration
import com.xebia.functional.xef.auto.ai
import com.xebia.functional.xef.auto.llm.openai.OpenAI
import com.xebia.functional.xef.pdf.pdf
import java.nio.file.Path

suspend fun main() {
  val userDir = System.getProperty("user.dir")
  val path = "$userDir/models/gpt4all/ggml-gpt4all-j-v1.3-groovy.bin"
  val url = huggingFaceUrl("orel12", "ggml-gpt4all-j-v1.3-groovy", "bin")
  val modelType = LLModel.Type.GPTJ
  val modelPath: Path = Path.of(path)
  val GPT4All = GPT4All(url, modelPath, modelType)

  println("🤖 GPT4All loaded: $GPT4All")

  val pdfUrl = "https://www.europarl.europa.eu/RegData/etudes/STUD/2023/740063/IPOL_STU(2023)740063_EN.pdf"

  /**
   * Uses internally [HuggingFaceLocalEmbeddings] default of "sentence-transformers", "msmarco-distilbert-dot-v5"
   * to provide embeddings for docs in contextScope.
   */

  ai {
    println("🤖 Loading PDF: $pdfUrl")
    contextScope(pdf(pdfUrl)) {
      println("🤖 Context loaded: $context")
      GPT4All.use { gpT4All: GPT4All ->
        println("🤖 Generating prompt for context")
        val prompt = gpT4All.promptMessage(
          "Describe in one sentence what the context is about.",
          promptConfiguration = PromptConfiguration {
            docsInContext(2)
          })
        println("🤖 Generating images for prompt: \n\n$prompt")
        val images =
          OpenAI.DEFAULT_IMAGES.images(prompt.joinToString("\n"), promptConfiguration = PromptConfiguration {
            docsInContext(1)
          })
        println("🤖 Generated images: \n\n${images.data.joinToString("\n") { it.url }}")
      }
    }
  }.getOrThrow()
}

This is all done locally, embeddings, context search and prompt:

gptj_model_load: loading model from '/Users/raulraja/workspace/xef/models/gpt4all/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx   = 2048
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 2
gptj_model_load: ggml ctx size = 5401.45 MB
gptj_model_load: kv self size  =  896.00 MB
gptj_model_load: ................................... done
gptj_model_load: model size =  3609.38 MB / num tensors = 285
🤖 GPT4All loaded: com.xebia.functional.gpt4all.GPT4All$Companion$invoke$1@6a024a67
18:35:20.363 [main] DEBUG ai.djl.util.cuda.CudaUtils -- cudart library not found.
18:35:20.365 [main] DEBUG a.d.h.tokenizers.jni.LibUtils -- Using cache dir: /Users/raulraja/.djl.ai/tokenizers/0.13.2-0.22.1-osx-aarch64
18:35:20.367 [main] DEBUG a.d.h.tokenizers.jni.LibUtils -- Loading huggingface library from: /Users/raulraja/.djl.ai/tokenizers/0.13.2-0.22.1-osx-aarch64
18:35:20.367 [main] DEBUG a.d.h.tokenizers.jni.LibUtils -- Loading native library: /Users/raulraja/.djl.ai/tokenizers/0.13.2-0.22.1-osx-aarch64/libtokenizers.dylib
🤖 Loading PDF: https://arxiv.org/pdf/2305.10601.pdf
🤖 Context loaded: com.xebia.functional.xef.vectorstores.CombinedVectorStore@5653d96c
🤖 Generating prompt for context
🤖 Generating images for prompt: 

["Human: The context is a conversation between two individuals, where one person asks for help with writing a document and the other provides guidance on how to do it."]
HttpClient: REQUEST: https://api.openai.com/v1/images/generations
METHOD: HttpMethod(value=POST)
COMMON HEADERS
-> Accept: application/json
-> Accept-Charset: UTF-8
-> Authorization: ***
CONTENT HEADERS
-> Content-Length: 245
-> Content-Type: application/json
HttpClient: RESPONSE: 200 OK
METHOD: HttpMethod(value=POST)
FROM: https://api.openai.com/v1/images/generations
COMMON HEADERS
-> CF-Cache-Status: DYNAMIC
-> CF-RAY: 7e084dada9e8314d-MAD
-> Connection: keep-alive
-> Content-Length: 545
-> Content-Type: application/json
-> Date: Sun, 02 Jul 2023 16:35:50 GMT
-> Server: cloudflare
-> access-control-allow-origin: *
-> alt-svc: h3=":443"; ma=86400
-> openai-organization: user-wwnpfgdsw5rs0l2qnxr90znl
-> openai-processing-ms: 7655
-> openai-version: 2020-10-01
-> strict-transport-security: max-age=15724800; includeSubDomains
-> x-request-id: 0eca056f3e19b3a8ea0664b93a4f5dec
🤖 Generated images: 

https://oaidalleapiprodscus.blob.core.windows.net/private/org-Iqvzi8lHN23LbvEwF91rsmhp/user-wWNpFGDsw5rs0l2qnxR90ZnL/img-Caz62h3gbllTLRMY7fNvljdF.png?st=2023-07-02T15%3A35%3A50Z&se=2023-07-02T17%3A35%3A50Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-07-01T20%3A10%3A11Z&ske=2023-07-02T20%3A10%3A11Z&sks=b&skv=2021-08-06&sig=XbmccNVLe3J9B9xmoVaV4EAJd58MtJydy6eGv6vlMog%3D

Process finished with exit code 0

[raulraja]

@xebia-functional/team-ai Ready for review

[diesalbla]

Added a few suggested changes .