Enable RAG flow with extensions without UI

[pmeier]

This addresses 3. from #16 (comment).

Imagine, we have a standalone ragna-core package, which is just the current backend, i.e. the "black box classes", with panel completely stripped out. From the technical side, how are we going to register any kind of plugins there?

• Right now registration is initiated by the CLI, but that is inherently tied to the UI and a library can't have a CLI.
• I intentionally also do not use hooks registered through entrypoints, i.e. registering a plugin by just installing a package. I feel this is a nice design for CLI tools that you invoke often like pytest, because you don't have also pass the extensions that you are using. But this is not the case here so I want this to be explicit.
• We also don't want to make the library stateful, i.e. provide a ragna_core.register_extension() function, that registers this globally.

One design that would be possible could look like this:

from ragna_core import Rag

# everything supported that we currently also support through CLI
extensions = [
    "ragna.extensions",
    "/path/to/my_custom/extension.py"
]

chat_params = {
    "llm_name": ...,
    "source_storage_name": ...,
    "chunk_size": ...,
    ...
}

# This is just for illustration purposes
# In practice, we probably want an abstraction layer around this 
# to enable generic reading and writing
documents = [
    "/path/to/my/document.pdf",
    ...
]

with Rag(*extensions) as rag:
    # input could be validated by pydantic
    # See 2. of https://github.com/Quansight/ragna/issues/16#issue-1864044081
    with session.new_chat(*documents, **chat_params) as chat:
        answer1 = chat.answer("What is Ragna?")
        answer2 = chat.answer("What is Quansight?")

cc @Adam-D-Lewis @costrouc

[pmeier]

The more I think about this, the more I'm convinced that this is the right way to do this. Thanks @costrouc for pushing hard here.

[dharhas]

I like this because it makes programatic usage much simpler and makes implementing workflows like what I just suggested in #18 fairly straightforward.

[costrouc]

Imagine, we have a standalone ragna-core package, which is just the current backend, i.e. the "black box classes", with panel completely stripped out.

I'd throw param into the list of things that shouldn't exist in ragna-core. But otherwise this is exactly what I'm asking for.

From the technical side, how are we going to register any kind of plugins there?

Yeah reading your reasoning on those three bullet points I get how we don't want automatic registration. And since we don't want automatic registration there really isn't much advantage over having a good base class and importing implementations.

To anticipate some base things I think we might be need. I know that many of these are already written.

• llm model base class
  • declares a schema for the llm configuration
• vector db base class
  • declares a schema for the encoding/chunking/vector db configuration
• document processing base class
  • should this detect the document type pdf/csv, etc.
  • opportunity to cache intermediate results (may want to store a document in several vector dbs)
• caching strategy base class (maybe/someday)
• validation strategy base class (maybe/someday)

But honestly as is the framework is already in a place that we could use.

I also think it is possible to have some higher level abstractions? Maybe this is out of scope but I think many of these actions would be quite common and I anticipate us needing them.

• calculate similarity of two messages
• categorize message and sentiment
• get llm response from set documents, message history
• process document into given vector db

[pmeier]

I'd throw param into the list of things that shouldn't exist in ragna-core. But otherwise this is exactly what I'm asking for.

Sure. My current plan is for ragna.core to only depend on pluggy and pydantic. The frontend would add a panel (and param) dependency while the backend would add fastapi.

I also think it is possible to have some higher level abstractions? Maybe this is out of scope but I think many of these actions would be quite common and I anticipate us needing them.

• calculate similarity of two messages

• categorize message and sentiment

These are analysis tasks. I feel this is out of scope for the orchestration layer. Discussed this with @Adam-D-Lewis yesterday. We will log the input and output of the LLM, so the information is there. This could be funneled into a tool like splunk for the analyst to access. Even if such a tool is not available, we also provide a hook to configure the logger:

ragna/ragna/_backend/hookspecs.py

Lines 28 to 30 in 9b73a79

	@hookspec
	def ragna_get_logger() -> Type[SourceStorage]:
	""""""

So users can put any kind of storing logic in there.

• get llm response from set documents, message history

Not sure if I understand this correctly, but isn't that exactly what chat.answer does above?

• process document into given vector db

That is a reasonable thing to have to give admins a way to manually perform maintenance tasks without going through the regular RAG workflow. Should be fairly simple to implement if we have the backend properly set up like I suggested above.

[costrouc]

Great I think we are on the same page 😄

[pmeier]

Working on this on the rewrite branch. I'll send a PR as soon as I get it into a presentable state.