How to get structured output from a query pipeline?

[SarahSohana27]

I have a pydantic class as below:

class CondensedQuestion(BaseModel):
        """Data model for new question."""
        new_user_question: str

how to add this output model to the llm prompt template

    # Next, we use the LLM to rewrite a user query
    rewrite = (
        "Please rewrite the user question using the current conversation.\n"
        "\n"
        "\n"
        "{chat_history}"
        "\n"
        "\n"
        "Latest message: {query}\n"
        # "Return your response as an output class CondensedQuestion."
    )
    rewrite_template = PromptTemplate(rewrite)

I want to get the new user question from the LLM as string and use it in the next component

[dosubot[bot]]

To get structured output from a query pipeline using a Pydantic model, you can integrate the model with the LLM prompt template by using an output parser. Here's how you can achieve this:

1. Define the Pydantic Model: You already have the CondensedQuestion model defined.

   class CondensedQuestion(BaseModel):
       """Data model for new question."""
       new_user_question: str

2. Create a Prompt Template: Use the PromptTemplate class to define your prompt.

   rewrite = (
       "Please rewrite the user question using the current conversation.\n"
       "\n"
       "\n"
       "{chat_history}"
       "\n"
       "\n"
       "Latest message: {query}\n"
   )
   rewrite_template = PromptTemplate(rewrite)

3. Use an Output Parser: Implement an output parser that can parse the LLM's output into your Pydantic model. You can use a custom output parser or a built-in one if available.

4. Integrate with the LLM: Use a program or function that ties the LLM, the prompt template, and the output parser together. This will ensure that the output from the LLM is parsed into the CondensedQuestion model.

Here's a conceptual example of how you might set this up:

from llama_index.program.lmformatenforcer import LMFormatEnforcerPydanticProgram
from llama_index.llms.llama_cpp import LlamaCPP

# Initialize the LLM
llm = LlamaCPP()

# Create a program that enforces the output format
program = LMFormatEnforcerPydanticProgram(
    output_cls=CondensedQuestion,
    prompt_template=rewrite_template,
    llm=llm,
    verbose=True,
)

# Execute the program with the necessary inputs
output = program(chat_history="Your chat history here", query="Your latest query here")

# Access the structured output
new_user_question = output.new_user_question

This setup will allow you to get the new user question as a structured output from the LLM, which you can then use in the next component of your query pipeline ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To integrate the coroutine arun_component with your query pipeline and avoid the "RuntimeWarning: coroutine 'QueryComponent.arun_component' was never awaited" error, you need to ensure that you are using the await keyword when calling arun_component. Here's how you can modify your query pipeline to handle this:

1. Ensure Asynchronous Execution: Make sure that the part of your code where you call arun_component is within an async function and that you use await to call it.

2. Modify Your Query Pipeline: If your query pipeline involves asynchronous components, you need to run the pipeline within an asynchronous context. Here's an example of how you might structure this:

import asyncio

# Define an async function to run your query pipeline
async def run_query_pipeline():
    # Assuming 'qp' is your query pipeline instance
    # and 'input_data' is the data you want to process
    result = await qp.run(input_data)
    print(result)

# Run the async function in an event loop
asyncio.run(run_query_pipeline())

3. Ensure All Components Support Asynchronous Execution: Check that all components in your pipeline that involve asynchronous operations are properly awaited. This includes any calls to arun_component or similar asynchronous methods.

By structuring your code this way, you ensure that all asynchronous operations are awaited, preventing the runtime warning and ensuring proper execution of your query pipeline.

To continue talking to Dosu, mention @dosu.

[SarahSohana27]

@dosu, how do you make a query pipeline async?

[dosubot[bot]]

To make a query pipeline asynchronous, you can use the await keyword with the arun method of the QueryPipeline object. This method is designed for asynchronous execution. Here's an example:

response = await p.arun(input="What did the author do during his time in YC?")

If you need to run multiple asynchronous operations, you can use the arun_multi method, which allows you to execute multiple components in parallel:

output = await p.arun_multi(
    {"qc1_0": {"input1": 1, "input2": 2}, "qc1_1": {"input1": 3, "input2": 4}}
)

These methods enable the query pipeline to execute asynchronously, potentially improving performance by running tasks in parallel ^[1][2].

To continue talking to Dosu, mention @dosu.

[SarahSohana27]

@dosu how to add :

# Create a program that enforces the output format
program = LMFormatEnforcerPydanticProgram(
    output_cls=CondensedQuestion,
    prompt_template=rewrite_template,
    llm=llm,
    verbose=True,
)

to a query pipeline. below is the query pipeline I have:

 qp = QP(
        modules={
            "router_input": InputComponent(),
            "rewrite_template": rewrite_template,
            "sllm": sllm,
            "sllm_response": rewrite_output_parser(),
            "router_component": router_c,
            "response_component": router_response(),
        },
        verbose=True,
    )

    # qp.add_link("router_input", "router_component", dest_key="input_str")
    qp.add_link("router_input", "rewrite_template", src_key="query", dest_key="query")
    qp.add_link(
        "router_input",
        "rewrite_template",
        src_key="chat_history",
        dest_key="chat_history",
    )
    qp.add_link("rewrite_template", "sllm")
    qp.add_link("sllm", "sllm_response")
    qp.add_link("sllm_response", "router_component", src_key="input_str", dest_key="input_str")
    qp.add_link("router_component", "response_component", dest_key="response")

    chat_engine = CustomChatEngine(qp, memory)

[dosubot[bot]]

To integrate the LMFormatEnforcerPydanticProgram into your existing query pipeline, you can replace the sllm module with the LMFormatEnforcerPydanticProgram. Here's how you can modify your query pipeline:

1. Initialize the LMFormatEnforcerPydanticProgram: Use the CondensedQuestion model and rewrite_template as you have already defined.

2. Replace the sllm Module: In your QP setup, replace the sllm module with the LMFormatEnforcerPydanticProgram instance.

Here's how you can do it:

# Create a program that enforces the output format
program = LMFormatEnforcerPydanticProgram(
    output_cls=CondensedQuestion,
    prompt_template=rewrite_template,
    llm=llm,  # Ensure this is your LLM instance
    verbose=True,
)

# Update the query pipeline
qp = QP(
    modules={
        "router_input": InputComponent(),
        "rewrite_template": rewrite_template,
        "sllm": program,  # Use the program here
        "sllm_response": rewrite_output_parser(),
        "router_component": router_c,
        "response_component": router_response(),
    },
    verbose=True,
)

# Define the links between components
qp.add_link("router_input", "rewrite_template", src_key="query", dest_key="query")
qp.add_link("router_input", "rewrite_template", src_key="chat_history", dest_key="chat_history")
qp.add_link("rewrite_template", "sllm")
qp.add_link("sllm", "sllm_response")
qp.add_link("sllm_response", "router_component", src_key="input_str", dest_key="input_str")
qp.add_link("router_component", "response_component", dest_key="response")

chat_engine = CustomChatEngine(qp, memory)

This setup integrates the LMFormatEnforcerPydanticProgram into your query pipeline, allowing it to enforce the output format as specified by your CondensedQuestion model.

To continue talking to Dosu, mention @dosu.