Fix agent scopes, and add documentation

[nomisRev]

Currently the agent blocks were updating the Atomic reference inside the AIScope, this mean that agents could change during the execution of such a block.

Since these blocks can be executed in parallel using parMap, or other concurrency operators. To prevent this I've scoped the lambdas to AIScope which create a child scope with an updated agent parameter. This way the blocks are truly scoped.

This PR also adds documentation to the current AI data type, and DSL.

Discussion / future work

This brings to question what to do with VectorStore, should an agent(..) { } block have it's own scoped VectorStore whilst also having access to the VectorStore of it's parent? We potentially have some kind-of factory to create VectorStore and compose them together by query'ing both and aggregating the results.

This brings to question to what to do with persistent vector stores like Postgres since all storage there is shared, and persisted across all ai calls and even multiple runs of the application. To avoid this we could create unique tables, which are destroyed at the end of an agent block but I guess that defeats the purpose of having a persisted vector store?

WDYT? @xebia-functional/team-ai

[nomisRev]

@raulraja I flattened the Agent / Tools types since from a technical point of view there currently wasn't having any benefit for having both for as far as I could tell.

[raulraja]

@raulraja I flattened the Agent / Tools types since, from a technical point of view, there currently wasn't any benefit for having both for as far as I could tell.

There is a distinction in libraries like langchain where agents have multiple running and execution policies, and that is the only difference. If we assume agents and tools are just functions and provide those behaviors through other means, such as retrying, and parallelizing, outside of the agent execution, then this is great.

[raulraja]

@nomisRev I think regarding the VectorStore I think the issue is that the interface itself is very restrictive and designed exclusively for the use case of finding similar documents. I think for it to be more useful, it would have to include more operators like get, put, search, and others as you would find in tools like Redis, Databases, and others; in general, I think it needs a more relaxed approach to become a key-value storage with search capabilities and that any tool can depend on. The tool could create or automatically have access to its own namespace beside other namespaces or tables.

We don't need to worry about combining, adding, or removing vector stores from contexts if we have something with these capabilities. The context that wants to be isolated would create its own table if needed or use the global memory.

This is related to the issue of long and short memory. Multiple agents may want to contribute to long-term memory for the entire ai block. In other use cases, agents or tools may want to have access to persistence but for a short period of time and then get rid of the memory when they are done. That ephemeral data while persisted or large may have only been relevant in the context of the agent run.

[raulraja]

thanks, @nomisRev, Looks great!

[juanpedromoreno]

Thanks @nomisRev !