notebooks/llmu/RAG_with_Quickstart_Connectors.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1lq03xVdfzMX"
      },
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/cohere-ai/notebooks/blob/main/notebooks/llmu/RAG_with_Quickstart_Connectors.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RHv_VYcmkRFI"
      },
      "source": [
        "# RAG with Quickstart Connectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZzk64vUhC8T"
      },
      "source": [
        "*Note: To run the notebook, you must first deploy your own Google Drive connector as a web-based REST API (the steps are outlined [here](https://txt.cohere.com/rag-quickstart-connectors/#build-and-deploy-the-connector)).*\n",
        "\n",
        "*The connector implementation code is [available here](examples/rag_quickstart_connectors/gdrive).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LgnnKlvWgfi8"
      },
      "source": [
        "This notebook shows how to build a RAG-powered chatbot with Cohere's Chat endpoint using one of 80+ pre-built quickstart connectors. We’ll use it to connect a chatbot to a Google Drive, enabling the chatbot to use the Google Drive API to find answers to a user’s question by searching documents in the Google Drive.\n",
        "\n",
        "Read the accompanying [article here](https://txt.cohere.com/rag-quickstart-connectors/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBb1kNk5gl9S"
      },
      "source": [
        "With quickstart connectors, developers can use pre-built implementations of over 80 connectors immediately without having to build them themselves. They can either use these quickstart connectors directly or adapt them to their organization’s needs.\n",
        "\n",
        "Cohere’s quickstart connectors are open-sourced and available in [our connectors repository](https://github.com/cohere-ai/quick-start-connectors/tree/main?ref=txt.cohere.com).\n",
        "\n",
        "The diagram below provides an overview of what we’ll build.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Workflow](../images/llmu/rag/rag-workflow-3.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ov6QwswXmfJ"
      },
      "source": [
        "# Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ofFN7iLvXn43",
        "outputId": "45f5bd8e-a76e-410e-f62f-4f96fb86c0fa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.2/117.2 kB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.8/77.8 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "! pip install cohere -q"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "P1wBvi56Rnmv"
      },
      "outputs": [],
      "source": [
        "import cohere\n",
        "from cohere import ChatConnector\n",
        "import uuid\n",
        "from typing import List\n",
        "\n",
        "co = cohere.Client(\"COHERE_API_KEY\") # Get your API key here: https://dashboard.cohere.com/api-keys"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "#@title Enable text wrapping in Google Colab\n",
        "\n",
        "from IPython.display import HTML, display\n",
        "\n",
        "def set_css():\n",
        "  display(HTML('''\n",
        "  <style>\n",
        "    pre {\n",
        "        white-space: pre-wrap;\n",
        "    }\n",
        "  </style>\n",
        "  '''))\n",
        "get_ipython().events.register('pre_run_cell', set_css)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IvtgPNskj5LY"
      },
      "source": [
        "# Test the Connector"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFCfGIZsk58k"
      },
      "source": [
        "We can test if the connector is working by making a curl request to its search endpoint.\n",
        "\n",
        "Note: The steps to build and deploy the connector are outlined in [this article](https://txt.cohere.com/rag-chatbot-quickstart/). The implementation code is available [here](https://github.com/cohere-ai/notebooks/tree/main/examples/chat_rag_quickstart_connector)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t8am2kZpcYpf",
        "outputId": "f316fbcc-6b77-4adf-b058-b198e0ba3039"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"results\": [\n",
            "    {\n",
            "      \"editedBy\": \"Meor Amer\",\n",
            "      \"id\": \"1LGsOhBL02jwy5UUIS8tuv9G80FSn7vxeQYiiglsN9oY\",\n",
            "      \"mimeType\": \"application/vnd.google-apps.document\",\n",
            "      \"modifiedTime\": \"2023-12-01T07:51:43.489Z\",\n",
            "      \"text\": \"\\ufeffConstructing Prompts\\r\\nIn this chapter, you'll learn about the different techniques for constructing prompts for the Command model.\\r\\nSuggest Edits\\r\\n!https://files.readme.io/76e9a4b-constructing-prompts-for-the-command-model.png\\r\\nIntroduction\\r\\nWhen working with large language models (LLMs), the prompt is the key to getting the desired response. A well-designed prompt will result in useful and accurate responses from a model and considerably improve your experience interacting with it.\\r\\nPrompts can be as simple as a one-liner, or they can be as complex as multiple layers of specific information. The more specific your command is, the more likely you will get exactly what you need from the model. We\\u2019ll look at some tips and ideas for constructing the commands in your prompt to help you get to your intended outcome. We\\u2019ll focus on the broad patterns without going into the long-tail list of techniques and tricks.\\r\\nWe\\u2019ll be using the Command model , Cohere\\u2019s instruction-following model that enables generative AI use cases in business productivity, marketing, creative writing, and more. This chapter comes with a Google Colaboratory notebook that lets you get hands-on with the code.\\r\\nSetting Up\\r\\nFirst, let\\u2019s install the Cohere Python SDK, get the Cohere API key, and set up the client.\\r\\nPython\\r\\n! pip install \"cohere<5\" import cohere co = cohere.Client(\\\"COHERE_API_KEY\\\") # Your Cohere API key\\r\\nLet\\u2019s also define a function to take a prompt and a temperature value and then call the Generate endpoint , which is how we can access the Command model. Here, we select the model type to be command.\\r\\nWe set a default temperature value of 0, which nudges the response to be more predictable and less random. Throughout this chapter, you\\u2019ll see different temperature values being used in different situations. Increasing the temperature value tells the model to generate less predictable responses and instead be more \\u201ccreative.\\u201d\\r\\nThis function returns the text response generated by the model.\\r\\nPython\\r\\ndef generate_text(prompt, temp=0): response = co.generate( model='command', prompt=prompt, max_tokens=200, temperature=temp) return response.generations[0].text\\r\\nInstruction\\r\\n!https://txt.cohere.com/content/images/2023/07/Instruction.png\\r\\nAdding basic instructions to a prompt\\r\\nWhile prompts can morph into something very lengthy and complex, it doesn\\u2019t have to be that way all the time. At its core, prompting a Command model is about sending an instruction to a text generation model and getting a response back. Hence, the smallest unit of a perfectly complete prompt is a short line of instruction to the model.\\r\\nLet\\u2019s say we want to generate a product description for a wireless headphone. Here\\u2019s an example prompt, where we create a variable for the user to input some text and merge that into the main prompt.\\r\\nPython\\r\\n`user_input = \\\"a wireless headphone product named the CO-1T\\\" prompt = f\\\"\\\"\\\"Write a creative product description for {user_input}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nThe model returns the following sample response, which does the job we asked for.\\r\\nThe CO-1T is a sleek and stylish wireless headphone product that offers a comfortable and secure fit for all-day wear. These headphones feature a noise-cancelling microphone and easy-to-use controls, making them perfect for use during your daily commute or while you're at the gym. The CO-1T comes with a charging cable and a carrying case, so you can take them with you on the go.\\r\\n\\r\\n\\r\\nThe CO-1T features a powerful and clear sound quality, making it easy to hear your favorite music and podcasts. The noise-cancelling microphone ensures that you can take calls without any distractions, while the easy-to-use controls allow you to adjust the volume and playback of your music.\\r\\n\\r\\n\\r\\nSpecifics\\r\\n!https://txt.cohere.com/content/images/2023/07/Specifics.png\\r\\nAdding specific details to a prompt\\r\\nA simple and short prompt can get you started, but in most cases, you\\u2019ll need to add specificity to your instructions. A generic prompt will return a generic response, and in most cases, that\\u2019s not what we want. In the same way that specific instructions will help humans do our job well, a model needs to be supplied with specific details to guide its response.\\r\\nGoing back to the previous prompt, the generated product description was great, but what if we wanted it to include specific things, such as its features, who it is designed for, and so on? We can adjust the prompt to take more inputs from the user, like so:\\r\\nPython\\r\\n`user_input_product = \\\"a wireless headphone product named the CO-1T\\\" user_input_keywords = '\\\"bluetooth\\\", \\\"wireless\\\", \\\"fast charging\\\"' user_input_customer = \\\"a software developer who works in noisy offices\\\" user_input_describe = \\\"benefits of this product\\\"\\r\\nprompt = f\\\"\\\"\\\"Write a creative product description for {user_input_product}, \\\\ with the keywords {user_input_keywords} for {user_input_customer}, and describe {user_input_describe}.\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nIn the example above, we pack the additional details of the prompt in a single paragraph. Alternatively, we can also compose it to be more structured, like so:\\r\\nPython\\r\\n`user_input_product = \\\"a wireless headphone product named the CO-1T\\\" user_input_keywords = '\\\"bluetooth\\\", \\\"wireless\\\", \\\"fast charging\\\"' user_input_customer = \\\"a software developer who works in noisy offices\\\" user_input_describe = \\\"benefits of this product\\\"\\r\\nprompt = f\\\"\\\"\\\"Write a creative product description for {user_input_product}. Keywords: {user_input_keywords} Audience: {user_input_customer} Describe: {user_input_describe}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nAnd here\\u2019s an example response. This time, the product description is tailored more specifically to our desired target customer, includes the key features that we specified, and sprinkles benefit statements throughout \\u2014 all coming from the instruction we added to the prompt.\\r\\nDo you hate noisy work environments? Well, we got a great solution for you! The CO-1T is the perfect wireless headphone for software developers who work in loud and disruptive offices. With its Bluetooth connectivity and noise-canceling features, you can stay focused on your work without any distractions.  Our product also has fast charging, so you won't have to worry about battery life. Instead, you can quickly get back to work in no time. So what are you waiting for? Become the most productive person in your office with the help of the CO-1T.\\r\\n\\r\\n\\r\\nThere are many other angles to add specificity to a prompt. Here are some examples:\\r\\n* Style : Telling the model to provide a response that follows a certain style or framework. For example, instead of asking the model to \\u201cGenerate an ad copy for a wireless headphone product\\u201d in the generic sense, we ask it to follow a certain style, such as \\u201cGenerate an ad copy for a wireless headphone product, following the AIDA Framework \\u2013 Attention, Interest, Desire, Action.\\u201d\\r\\n* Tone : Adding a line mentioning how the tone of a piece of text should be, such as professional, inspirational, fun, serious, and so on. For example, \\u201cTone: casual\\u201d\\r\\n* Persona : Telling the model to act like a certain persona helps to add originality and quality to the response. For example, \\u201cYou are a world-class content marketer. Write a product description for\\u2026\\u201d\\r\\n* Length : Telling the model to generate text with a specific length, be it in words, paragraphs, and others. This helps guide the model to be verbose, concise, or somewhere in between. For example, \\u201cWrite in three paragraphs the benefits of \\u2026\\u201d\\r\\nContext\\r\\n!https://txt.cohere.com/content/images/2023/07/Context.png\\r\\nAdding contextual information to a prompt\\r\\nWhile LLMs excel in text generation tasks, they struggle in context-aware scenarios. Here\\u2019s an example. If you were to ask the model for the top qualities to look for in wireless headphones, it will duly generate a solid list of points. But if you were to ask it for the top qualities of the CO-1T headphone, it will not be able to provide an accurate response because it doesn\\u2019t know about it (CO-1T is a hypothetical product we just made up for illustration purposes).\\r\\nIn real applications, being able to add context to a prompt is key because this is what enables personalized generative AI for a team or company. It makes many use cases possible, such as intelligent assistants, customer support, and productivity tools, that retrieve the right information from a wide range of sources and add it to the prompt.\\r\\nThis is a whole topic on its own, but to provide some idea, this demo shows an example of text generation with information retrieval in action. Here though, we\\u2019ll assume that the right information is already retrieved and added to the prompt.\\r\\nHere\\u2019s an example where we ask the model to list the features of the CO-1T wireless headphone without any additional context:\\r\\nPython\\r\\n`user_input =\\\"What are the key features of the CO-1T wireless headphone\\\" prompt = user_input\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nThis generates a response that the model makes up since it doesn\\u2019t have any information to refer to.\\r\\nThe CO-1T wireless headphone is a high-quality, comfortable, and durable headphone that is designed for use with a variety of devices. It features a sleek and modern design, a comfortable and secure fit, and a high-quality sound. The CO-1T is also equipped with a variety of features, including a built-in microphone, a multi-function button, and a rechargeable battery.\\r\\n\\r\\n\\r\\nAnd here\\u2019s the same request to the model, this time with the product description of the product added as context.\\r\\nPython\\r\\n`context = \\\"\\\"\\\"Think back to the last time you were working without any distractions in the office. That's right...I bet it's been a while. \\\\ With the newly improved CO-1T noise-cancelling Bluetooth headphones, you can work in peace all day. Designed in partnership with \\\\ software developers who work around the mayhem of tech startups, these headphones are finally the break you've been waiting for. With \\\\ fast charging capacity and wireless Bluetooth connectivity, the CO-1T is the easy breezy way to get through your day without being \\\\ overwhelmed by the chaos of the world.\\\"\\\"\\\"\\r\\nuser_input = \\\"What are the key features of the CO-1T wireless headphone\\\"\\r\\nprompt = f\\\"\\\"\\\"{context} Given the information above, answer this question: {user_input}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nNow, the model accurately lists the features of the model.\\r\\nThe answer is:\\r\\nThe CO-1T wireless headphones are designed to be noise-canceling and Bluetooth-enabled. They are also designed to be fast charging and have wireless Bluetooth connectivity.\\r\\n\\r\\n\\r\\nFormat\\r\\n!https://txt.cohere.com/content/images/2023/07/Format.png\\r\\nAdding output format requirements to a prompt\\r\\nSo far, we have seen how to get the model to generate responses that follow certain styles or include specific information. But we can also get the model to generate responses in a certain format. Let\\u2019s look at a couple of them: markdown tables and JSON strings.\\r\\nHere, the task is to extract information from a list of invoices. Instead of providing the information in plain text, we can prompt the model to generate a table containing all the required information.\\r\\nPython\\r\\n`prompt=\\\"\\\"\\\"Turn the following information into a table with columns Invoice Number, Merchant Name, and Account Number. Bank Invoice: INVOICE #0521 MERCHANT ALLBIRDS ACC XXX3846 Bank Invoice: INVOICE #6781 MERCHANT SHOPPERS ACC XXX9877 Bank Invoice: INVOICE #0777 MERCHANT CN TOWER ACC XXX3846 \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nThe response will come in the form of a markdown table.\\r\\n| Invoice Number | Merchant Name | Account Number |\\r\\n|-----------|------------|-----------|\\r\\n| 0521 | Allbirds | XXX3846 |\\r\\n| 6781 | Shoppers | XXX9877 |\\r\\n| 0777 | CN Tower | XXX3846 |\\r\\n\\r\\n\\r\\nAnother useful format is JSON, which we can modify the prompt as follows.\\r\\nPython\\r\\n`prompt=\\\"\\\"\\\"Turn the following information into a JSON string with the following keys: Invoice Number, Merchant Name, and Account Number. Bank Invoice: INVOICE #0521 MERCHANT ALLBIRDS ACC XXX3846 Bank Invoice: INVOICE #6781 MERCHANT SHOPPERS ACC XXX9877 Bank Invoice: INVOICE #0777 MERCHANT CN TOWER ACC XXX3846 \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nThis returns the following response.\\r\\nJSON\\r\\n[ { \\\"Invoice Number\\\": \\\"0521\\\", \\\"Merchant Name\\\": \\\"Allbirds\\\", \\\"Account Number\\\": \\\"XXXX3846\\\" }, { \\\"Invoice Number\\\": \\\"6781\\\", \\\"Merchant Name\\\": \\\"Shoppers\\\", \\\"Account Number\\\": \\\"XXXX9877\\\" }, { \\\"Invoice Number\\\": \\\"0777\\\", \\\"Merchant Name\\\": \\\"CN Tower\\\", \\\"Account Number\\\": \\\"XXXX3846\\\" } ]\\r\\nExamples\\r\\n!https://txt.cohere.com/content/images/2023/07/Examples.png\\r\\nAdding examples to a prompt\\r\\nAll our prompts so far use what is called zero-shot prompting , which means that we are providing instruction without any example. But in many cases, it is extremely helpful to provide examples to the model to guide its response. This is called few-shot prompting .\\r\\nFew-shot prompting is especially useful when we want the model response to follow a certain style or format. Also, sometimes it is hard to explain what you want in an instruction, and easier to show examples.\\r\\nLet\\u2019s use an example task, where a model should take a request coming from a human and rephrase it into the most accurate utterance that an AI virtual assistant should use. The example data is taken from this paper (Einolghozati et al., 2020).\\r\\nWe\\u2019ll use this example request: \\u201cSend a message to Alison to ask if she can pick me up tonight to go to the concert together\\u201d. Given that request, we should expect the rephrased utterance to be something like: \\u201cCan you pick me up tonight to go to the concert together?\\u201d\\r\\nFirst, let\\u2019s generate a response without giving the model an example. Here\\u2019s the prompt:\\r\\nPython\\r\\n`prompt=\\\"\\\"\\\"Turn the following message to a virtual assistant into the correct action: Send a message to Alison to ask if she can pick me up tonight to go to the concert together\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nThe response we get is not wrong, but it doesn\\u2019t follow the style that we need, which is a simple one-line rephrasing of the original request. Instead, it generates an email!\\r\\nHere is the message to Alison:\\r\\n\\r\\n\\r\\nHey Alison, I hope you're doing well! I was wondering if you could pick me up tonight to go to the concert together. I would really appreciate it, and I think it would be a lot of fun. Let me know if you're able to do this, and I'll make sure to be ready on time.\\r\\n\\r\\n\\r\\nThanks,\\r\\n[Your Name]\\r\\n\\r\\n\\r\\nNow, let\\u2019s modify the prompt by adding a few examples of how we expect the output to be.\\r\\nPython\\r\\n`user_input = \\\"Send a message to Alison to ask if she can pick me up tonight to go to the concert together\\\"\\r\\nprompt=f\\\"\\\"\\\"Turn the following message to a virtual assistant into the correct action:\\r\\nMessage: Ask my aunt if she can go to the JDRF Walk with me October 6th Action: can you go to the jdrf walk with me october 6th\\r\\nMessage: Ask Eliza what should I bring to the wedding tomorrow Action: what should I bring to the wedding tomorrow\\r\\nMessage: Send message to supervisor that I am sick and will not be in today Action: I am sick and will not be in today\\r\\nMessage: {user_input}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nThis time, the style of the response is exactly how we want it.\\r\\nCan you pick me up tonight to go to the concert together?\\r\\n\\r\\n\\r\\nChain of Thought\\r\\nOne specific way to provide examples in a prompt is to show responses that include a reasoning step. This way, we are asking the model to \\u201cthink\\u201d first rather than going straight to the response. In tasks involving mathematical questions, for example, there is a huge difference between directly giving the answer and adding a reasoning step in between.\\r\\nThis concept is called chain of thought prompting, introduced by Wei et al. Let\\u2019s look at an example from the paper which illustrates this idea.\\r\\nFirst let\\u2019s look at a prompt without a chain of thought. It contains one example of a question followed by the answer, without any intermediate calculation step. It also contains the new question we want to answer.\\r\\nPython\\r\\n`prompt=f\\\"\\\"\\\" Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. \\\\ How many tennis balls does he have now? A: The answer is 11. Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? A:\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nWe get the following response, which is an incorrect answer. And notice that the response is direct, in the same style as the example given.\\r\\nThe answer is 29.\\r\\n\\r\\n\\r\\nNow, let\\u2019s repeat that, this time with a chain of thought. Now, the example answer contains a reasoning step, describing the calculation logic to get to the final answer, before giving the final answer.\\r\\nPython\\r\\n`prompt=f\\\"\\\"\\\" Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. \\\\ How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. \\\\ The answer is 11. Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? A:\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nAnd we get the correct answer this time, with the response following the style of the example given.\\r\\nThe cafeteria started with 23 apples. They used 20 to make lunch, so they have 23 - 20 = 3 apples. They bought 6 more apples, so they have 3 + 6 = 9 apples. The answer is 9.\\r\\n\\r\\n\\r\\nSteps\\r\\n!https://txt.cohere.com/content/images/2023/07/Steps.png\\r\\nAdding generation steps to a prompt\\r\\nTo steer the model toward generating higher-quality responses, it can be helpful to add instructions for the model to generate intermediate steps before generating the final output. The information generated during these steps helps enrich the model\\u2019s context before it generates the final response.\\r\\nThere could be another scenario where we specifically need the response to be detailed and verbose. In this case, asking the model to generate responses in steps will become handy.\\r\\nLet\\u2019s use an example of generating startup ideas. We can get the model to directly generate an idea for a given industry, like so:\\r\\nPython\\r\\n`user_input = \\\"education\\\"\\r\\nprompt = f\\\"\\\"\\\"Generate a startup idea for this industry: {user_input}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nThis generates the following response, which is reasonable, but perhaps not rich enough in information.\\r\\nA mobile app that connects students with tutors for on-demand homework help.\\r\\n\\r\\n\\r\\nAlternatively, we can ask the model to generate information in steps, such as describing the problem to be solved and the target audience experiencing this problem.\\r\\nPython\\r\\n`user_input = \\\"education\\\"\\r\\nprompt = f\\\"\\\"\\\"Generate a startup idea for this industry: {user_input} First, describe the problem to be solved. Next, describe the target audience of this startup idea. Next, describe the startup idea and how it solves the problem for the target audience. Next, provide a name for the given startup.\\r\\nUse the following format: Industry: <the given industry> The Problem: <the given problem> Audience: <the given target audience> Startup Idea: <the given idea> Startup Name: <the given name>\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.9) print(response)`\\r\\nThis provides a richer description of the startup idea.\\r\\nIndustry: Education\\r\\nThe Problem: Students often need to learn at their own pace and require individual attention to succeed in their studies.\\r\\nAudience: Students who need a more personalized learning experience.\\r\\nStartup Idea: An online platform that connects students with tutors for personalized one-on-one learning.\\r\\nStartup Name: tutormate\\r\\n\\r\\n\\r\\nPrefix\\r\\n!https://txt.cohere.com/content/images/2023/07/Prefix.png\\r\\nAdding a prefix to a prompt\\r\\nTo ensure that the response follows a consistent format or style, sometimes we need to add a prefix or leading words to help guide the response. This is especially handy when the temperature value is high. In this scenario, we want the response to be creative but, at the same time, follow a certain format.\\r\\nLet\\u2019s say we are generating the characteristics of football players for a given position, with one separate paragraph per characteristic. A prompt without any guiding prefix would look something like this:\\r\\nPython\\r\\n`user_input_position = \\\"modern centre forward\\\"\\r\\nprompt = f\\\"\\\"\\\"Describe the ideal {user_input_position}. In particular, describe the following characteristics: \\\\ pace, skill, and awareness.\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.9) print(response)`\\r\\nAnd the response is a paragraph combining all the characteristics \\u2014 not what we wanted.\\r\\nAn ideal modern centre forward would have a combination of pace, skill, and awareness. They would be fast and agile, with the ability to run and move with the ball at speed. They would also be skilled with their feet, with the ability to dribble and pass the ball with ease. Finally, they would be aware of their surroundings and the position of their teammates and opponents, with the ability to make quick and intelligent decisions on the fly.\\r\\n\\r\\n\\r\\nBut if we just added a prefix of the first characteristic (\\u201cPace\\u201d) at the end of the prompt, it will give a signal to the model as to how the output should look like.\\r\\nPython\\r\\n`user_input_position = \\\"modern centre forward\\\"\\r\\nprompt = f\\\"\\\"\\\"Describe the ideal {user_input_position}. In particular, describe the following characteristics: \\\\ pace, skill, and awareness.\\r\\nPace:\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.9) print(response)`\\r\\nHere\\u2019s an example response, which is much closer to what we are looking for:\\r\\nThe ideal modern centre forward will have excellent pace, both in terms of speed and acceleration. This will allow them to take advantage of gaps in the opposition defence and create chances for their team.\\r\\nSkill:\\r\\n\\r\\n\\r\\nThe ideal modern centre forward will also have a high level of skill, including dribbling, passing, and shooting. This will allow them to create chances for themselves and their team, as well as to score goals.\\r\\n\\r\\n\\r\\nAwareness:\\r\\n\\r\\n\\r\\nThe ideal modern centre forward will also have a high level of awareness, including an understanding of the game and their position within it. This will allow them to make intelligent decisions on the pitch, both in terms of their own play and in terms of the overall strategy of their team.\\r\\n\\r\\n\\r\\nZero-Shot Chain of Thought\\r\\nOne specific way of adding prefixes to a prompt is such that it encourages the model to perform a reasoning step before generating the final answer.\\r\\nEarlier, we saw that chain-of-thought prompting helps guide the model to perform reasoning by way of showing examples. But Kojima et al. show that there\\u2019s a way to apply chain-of-thought prompting without providing examples. This technique is called zero-shot chain of thought In contrast, the technique with examples discussed earlier is known as \\u201cfew-shot chain of thought\\u201d.\\r\\nThe paper proposes adding a prefix that nudges the model to perform a reasoning step, specifically the phrase \\u201cLet\\u2019s think step by step\\u201d.\\r\\nHere\\u2019s an example taken from the paper. First, we look at a prompt without the \\u201cLet\\u2019s think step by step\\u201d prefix.\\r\\nPython\\r\\n`prompt=f\\\"\\\"\\\"Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there? A: The answer (arabic numerals) is\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nIt gives an incorrect answer.\\r\\n8\\r\\n\\r\\n\\r\\nNow, let\\u2019s add the \\u201cLet\\u2019s think step by step\\u201d prefix to the prompt.\\r\\nPython\\r\\n`prompt=f\\\"\\\"\\\"Q: A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there? A: Let\\u2019s think step by step.\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nAnd this time, the response contains a reasoning step before giving the final answer, which is the correct answer.\\r\\nThere are 16 balls in total. Half of the balls are golf balls, so there are 8 golf balls. Half of the golf balls are blue, so there are 4 blue golf balls.\\r\\nSo, the answer is 4.\\r\\n\\r\\n\\r\\nConclusion\\r\\nIn this chapter, we looked at some techniques for constructing prompts for the Command model. A prompt can be as simple as a single line of instruction, though the more specific the prompt is, the higher the quality and accuracy you can expect from the response. Each building block added to a prompt provides a different type of lever to enhance the quality of the response.\",\n",
            "      \"title\": \"Constructing Prompts\",\n",
            "      \"url\": \"https://docs.google.com/document/d/1LGsOhBL02jwy5UUIS8tuv9G80FSn7vxeQYiiglsN9oY/edit?usp=drivesdk\"\n",
            "    },\n",
            "    {\n",
            "      \"editedBy\": \"Meor Amer\",\n",
            "      \"id\": \"1oF20QD0lHNdYQp6F7sSyEC1grGErout4GIQn1JBUACo\",\n",
            "      \"mimeType\": \"application/vnd.google-apps.document\",\n",
            "      \"modifiedTime\": \"2023-12-01T07:50:28.605Z\",\n",
            "      \"text\": \"\\ufeffChaining Prompts\\r\\nIn this chapter, you'll learn about several prompt-chaining patterns and their example applications.\\r\\nSuggest Edits\\r\\n!https://files.readme.io/17ec857-Chaining_Prompts.png\\r\\nIntroduction\\r\\nIn the previous two chapters, we looked at various ways to prompt the Command model. One thing similar about all of those examples is that they utilize a single prompt to fulfill a particular task. However, there may be other tasks where this is insufficient, and we instead have to chain a few prompts together to complete a task successfully.\\r\\nWe can think of the previous two chapters as looking at prompts as a single unit, and this chapter will look at prompts as a combination of these units. The diagram below summarizes these three chapters.\\r\\n!https://files.readme.io/4f17dcc-chapters_1-3_comparison.png\\r\\nA comparison of this chapter and the two previous ones\\r\\nCombining multiple prompts to accomplish a single goal will be the focus of this chapter. We\\u2019ll look at some ideas and patterns for chaining prompts, which could come in handy when building a generative AI application.\\r\\nSequential\\r\\nLet\\u2019s begin with probably the most common approach for chaining prompts \\u2013 in a sequence.\\r\\n!https://txt.cohere.com/content/images/2023/08/Sequential.png\\r\\nChaining prompts in a sequence\\r\\nThe key idea is that if the task provided in a single prompt becomes too lengthy or complex and contains many different instructions, the response may not capture the details and granularity required. In this case, it would make sense to break this task into smaller subtasks that can be run one after the other. The response of one subtask becomes the prompt to the other subtask, and this chain continues until the overall task is complete.\\r\\nLet's use story generation as an example. Suppose we have a task to generate a long story containing a dialog guided by a set of information \\u2013 characters, story beats, and locations. Now, we could stuff all this information into a single prompt, but that might dilute the key details we want the dialog to contain.\\r\\nBesides, we don\\u2019t want to write all these additional details in the first place because we\\u2019d like the model to generate them for us. What we want is to provide a short summary of what we want the story to be about, which becomes our sole prompt.\\r\\nThe diagram below summarizes the chains involved in generating the eventual dialog. First, a human enters a story summary, which becomes the prompt to generate a list of characters, which then becomes the prompt to generate the story beats, and so on, until we get to the dialog generation.\\r\\n!https://txt.cohere.com/content/images/2023/08/sequential-example.png\\r\\nA summary of the different steps in the story generation task, running in a sequence\\r\\nAnother added benefit of chaining prompts in sequences instead of using one single prompt is that we can implement human checks at each point in the sequence, and if a response does not fulfill a certain standard, we can trigger a re-run. This makes it easy to do evaluation and quality control over a complex generative AI task \\u2013 by doing them in stages.\\r\\nParallel\\r\\nA sequential chain of prompts is needed when the subtasks depend on each other. But when they are independent, we can run them in parallel instead.\\r\\n!https://txt.cohere.com/content/images/2023/08/Parallel.png\\r\\nRunning prompts in parallel before combining them into a single output\\r\\nLet\\u2019s say we are building an application that generates recipe ideas for a whole week and then generates a shopping list of ingredients for the user to buy. In this case, given a user input of, say, the number of meals or days, we can run the recipe generation step in parallel. The prompt might look something like the following:\\r\\nPython\\r\\nprompt = f'Suggest a simple and quick recipe for {meal}. Write in JSON containing these keys \\\"Ingredients\\\" and \\\"Instructions\\\"'\\r\\nNext, we\\u2019ll repeat the recipe generation across all meals. Once complete, we can consolidate the ingredients from each meal into a single shopping list that the user can use immediately.\\r\\nPython\\r\\nprompt = f\\\"\\\"\\\"Consolidate the following ingredients into a single shopping list, without repetition: {ingredients}\\\"\\\"\\\"\\r\\nSampling\\r\\nIf a task involves logical reasoning, for example, in mathematical question answering, there may be more than one way to solve a given problem. In this case, we want to encourage the model to be more creative when exploring different solutions. But dialing up the creativity knob (i.e., \\u201ctemperature\\u201d) also means that there\\u2019s a higher chance that the model will get the final answer wrong.\\r\\nThe solution is, instead of prompting the same question to the model once, we can repeat that multiple times and choose the response with the highest majority.\\r\\n!https://txt.cohere.com/content/images/2023/08/Sampling.png\\r\\nSampling prompt responses followed by a majority vote\\r\\nLet\\u2019s look at an example taken from a paper by Wang et al., 2023 that introduces the concept of self -consistency .\\r\\nFirst, revisiting the previous chapter , we looked at the concept of chain-of-thought prompting introduced by Wei et. al, 2023 , where a model is prompted in such a way that it is encouraged to do a reasoning step before giving the final response. In those settings, however, the model is typically encouraged to do \\u201cgreedy decoding,\\u201d which means biasing towards the correct and safe path. This can be done by adjusting settings like the temperature value.\\r\\nWith self-consistency, we can build on the chain-of-thought approach by sampling from several paths instead of one. We also make the paths much more diverse by adjusting the settings towards being more \\u201ccreative,\\u201d again using settings like temperature. We then do a majority vote out of all answers.\\r\\nThe diagram below illustrates the self-consistency concept. It shows an example of comparing the result of a single solution, using the greedy decoding approach, and a solution that samples from multiple generations, using a more diverse decoding approach. The former didn\\u2019t get the answer correct, and the latter did.\\r\\nhttps://lh6.googleusercontent.com/c4nCCvs8OfEJl-z3eOQ6-C7ypOwYc-84f0Hfk3aMdJjbxVLuVU9AK0ZmTp4nAfj-E_woIlTBu1kJMvbE8JrKuSAYEcgwgxAhvbWttrHmLB0xNer5ug7k-wlnUC_KYIVpDg5yVTmHWvZcmRy4O36F9-E\\r\\nAn example from Wei et al., 2023 illustrating the self-consistency concept\\r\\nExploration\\r\\nBuilding on the previous chain pattern, when a question is much more complex and requires more granular reasoning steps, it is helpful to break down the steps into pieces. At each step, we encourage the model to explore different solutions, vote for the best solution and only then continue to the next step.\\r\\nThis is also useful in creative applications like story writing. Making the model explore different story ideas is more interesting than constraining it to a single path, elevating the final output's quality.\\r\\n!https://txt.cohere.com/content/images/2023/08/Exploration.png\\r\\nExploring different solutions, one step at a time\\r\\nLet\\u2019s use one concrete example from Yao et al., 2023 , which introduces the concept of tree of thoughts . The diagram below illustrates how this concept differs from direct input-output prompting, chain-of-thought, and self-consistency with chain-of-thought. At each step, the model generates several diverse solutions. A separate prompt is then used to evaluate these solutions and vote for the best solution. The process repeats until the final step is completed.\\r\\nhttps://lh6.googleusercontent.com/nI7-VerhzDlQiqtzdy-JujxQULEktPFrqlek_VrwaHnZVlEUEtFpo7yTHRYi5Ai0vubp0yi6tFT1e_PX9DajfRPh_0O0p-oKLn0cs9Jtbg166rM4Z-ZmzGDXgf0Uy1j9NBw7ha2lYdB8mydqNfWg13w\\r\\nAn illustration from Yao et al., 2023 of the Tree of Thoughts approach compared to three others\\r\\nHere\\u2019s one of the examples shown in the paper with a creative writing task. The diagram below shows the initial input containing a list of four seemingly unrelated sentences \\u2013 each talking about handstands, the smell of space, sign language, and people\\u2019s perceptions. The task is to weave them into one coherent passage. It must contain four short paragraphs, each ending with the given sentence.\\r\\nhttps://lh4.googleusercontent.com/oW9iUE6aWqaxb4p26wmji6MO9yAmDH_-4renhnIiJTJbe9QXg6CZl_detvcMG-5Rf7kepegWvjplZi5PzrRuEz7GMrENToB3IVr17VMPnRSCCR0GDwyS7AkmF56n56uWyob-yTD8WjkZmI_sSIpCp8g\\r\\nA creative writing example from Yao et al., 2023 using Tree of Thoughts\\r\\nSince the task is quite challenging, it makes sense to break down the task into two: writing the plan and writing the actual passage based on the winning plan. At each step, the model generates a few solutions, and then another prompt is used to evaluate and vote for the best solution, guiding the direction of the next step.\\r\\nA couple of writing plan options are shown in the screenshot, and the winning option is the one that suggests using self-help as the theme to weave the sentences into a coherent passage.\\r\\nLoop\\r\\nIn some applications, we may need to re-run a generation step, given what happens in a subsequent step. One example is when that subsequent step is used to check if the response generated meets specific criteria, such as quality and format. This is where the loop pattern becomes useful.\\r\\n!https://txt.cohere.com/content/images/2023/08/Loop.png\\r\\nLooping a task back to an earlier step\\r\\nLet\\u2019s take a rephrasing task as an example. Say we have an application that takes a rude user comment and suggests a rephrasing that makes it more polite. Here, we want the LLM response to be polite and, at the same time, retain the original meaning of the comment.\\r\\nThe prompt, taking in the user input to be rephrased, might look something like the following:\\r\\nPython\\r\\n`user_input = \\\"I really don't have time for this nonsense.\\\"\\r\\nprompt_rephrase = f\\\"\\\"\\\"Rephrase this user comment into something more polite: User comment: You don't know what you're talking about. Rephrased comment: I think there might be some inaccuracies in your statement. User comment: {user_input} Rephrased comment:\\\"\\\"\\\"`\\r\\nHere is an example response which rephrases the original user input:\\r\\nI think we might need to set aside some time to discuss this properly.\\r\\n\\r\\n\\r\\nNext, we create another prompt to check if the rephrased comment is similar enough to the original comment.\\r\\nPython\\r\\n`prompt_check = f\\\"\\\"\\\"Below is a rude comment that has been rephrased into a polite version. The rephrased comment must maintain a similar meaning to the original comment. Check if this is true. Answer with YES or NO.\\r\\nOriginal comment: Shut up, you're always wrong. Rephrased comment: Please be quiet, check your facts again. Similar meaning: YES Original comment: I can't stand you. Rephrased comment: Let's discuss this. Similar meaning: NO Original comment: {user_input} Rephrased comment: {user_input_rephrased} Similar meaning:\\\"\\\"\\\"`\\r\\nAnd if the response is \\u201cNO,\\u201d we route the task back to the rephrasing prompt and repeat until we get a \\u201cYES\\u201d response.\\r\\nPerformance Considerations\\r\\nPrompt chaining is a powerful concept that makes complex use cases possible when a single prompt setup is insufficient. Having said that, prompt chaining should only be considered when it\\u2019s truly necessary. The overall performance considerations need to be taken into account.\\r\\nOne such consideration is latency. The longer the chain, the longer it takes to complete a task from start to finish. If an application is latency-sensitive, it makes sense to minimize the number of chains. The cost factor is another consideration when designing applications that rely on prompt chaining.\\r\\nFinal Thoughts\\r\\nIn this chapter, we looked at several prompt-chaining patterns and examples of how they can be applied to the Command model.\\r\\nThis is a fascinating area of prompt engineering because it opens up so much room for creativity when solving problems with LLMs. Although it comes with some performance trade-offs, balancing these considerations presents an exciting challenge for building production-ready LLM-powered applications.\",\n",
            "      \"title\": \"Chaining Prompts\",\n",
            "      \"url\": \"https://docs.google.com/document/d/1oF20QD0lHNdYQp6F7sSyEC1grGErout4GIQn1JBUACo/edit?usp=drivesdk\"\n",
            "    },\n",
            "    {\n",
            "      \"editedBy\": \"Meor Amer\",\n",
            "      \"id\": \"1X9XM5nLjjF_8b6ulj35FK8ioWCJTg6n0KtYtXLkr9Dw\",\n",
            "      \"mimeType\": \"application/vnd.google-apps.document\",\n",
            "      \"modifiedTime\": \"2023-12-01T07:51:15.475Z\",\n",
            "      \"text\": \"\\ufeffUse Case Patterns\\r\\nIn this chapter, you'll learn about the common use case patterns in text generation, including prompt examples.\\r\\nSuggest Edits\\r\\n!https://files.readme.io/c798ac6-command-usecase-patterns.png\\r\\nIn the previous chapter, we took the vertical direction and learned different ways to construct a prompt. In this chapter, we'll take a horizontal approach by looking at different use case patterns for applying these prompts.\\r\\nIn this chapter, we'll go through several use case patterns for the Command model. Though they won\\u2019t cover all the possible ways you can use the model, they are good starting points for understanding the patterns of tasks where the model works well.\\r\\nSetting Up\\r\\nThe examples in this post will be shown in Python. For each use case, we\\u2019ll look at ideas on how a prompt can be constructed and the associated model settings. This chapter comes with a Google Colaboratory notebook that lets you get hands-on with the code.\\r\\nFirst, install the Cohere package, get the Cohere API key , and set up the client.\\r\\n! pip install cohere\\r\\n\\r\\n\\r\\nimport cohere\\r\\nco = cohere.Client(\\\"COHERE_API_KEY\\\") # Your Cohere API key\\r\\n\\r\\n\\r\\nLet\\u2019s also define a function to take a prompt and a temperature value and then call the Generate endpoint , which is how we can access the Command model. Here, we select the model to be command . We set a default temperature value of 0, which nudges the response to be more predictable and less random. This function returns the text response generated by the model.\\r\\nPython\\r\\ndef generate_text(prompt, temp=0): response = co.generate( model='command', prompt=prompt, max_tokens=300, temperature=temp) return response.generations[0].text\\r\\nOur examples will revolve around a company\\u2019s activities for launching a new wireless headphone product, such as getting the word out, managing customer interactions, and so on. For this, let\\u2019s define a text snippet containing the product description. We\\u2019ll be utilizing this snippet in several examples throughout this chapter.\\r\\nPython\\r\\nproduct=\\\"\\\"\\\"The CO-1T is a wireless headphone product that uses Bluetooth technology to connect to your devices. \\\\\\\\ It has a long battery life and can be quickly charged using the included USB cable. The headphone is \\\\\\\\ lightweight and comfortable, ideal for long periods of use. It has a built-in microphone for making calls, \\\\\\\\ and a button to control the volume. The CO-1T is a great choice for anyone looking for a wireless headphone \\\\\\\\ product with great battery life.\\\"\\\"\\\"\\r\\nWriting\\r\\n!https://files.readme.io/e7fd8cf-writing.png\\r\\nWe\\u2019ll start with the most general type of use case: writing.\\r\\nHere we can ask the model to write freeform text, for example, with this prompt: \\u201cCreate an email about the launch of the wireless headphone product.\\u201d But that alone might not be very useful in practical applications because the generated text can go in very different directions. We may want to add more context and specificity to the prompt.\\r\\nHere\\u2019s how we can do that. Let\\u2019s say we\\u2019re building an application for users to enter some bullet points and get a complete email written. We can set up the prompt in the following way: create a variable for the user to input some text and merge that, together with the product description, into the main prompt.\\r\\nPython\\r\\n`user_input =\\\"\\\"\\\"\\r\\n* announce product launch\\r\\n* create a call to action\\r\\n* mention live chat for support \\\"\\\"\\\"\\r\\nprompt = f\\\"\\\"\\\"{product} Create an email about the product above mentioning the following: {user_input} \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nAdding this context helps to guide the model in the direction we want it to go. Here is a sample response:\\r\\nHello,\\r\\n\\r\\n\\r\\nWe are excited to announce the launch of our new wireless headphone product, the CO-1T. This product uses Bluetooth technology to connect to your devices and has a long battery life. It is lightweight and comfortable, making it ideal for long periods of use. The CO-1T also has a built-in microphone for making calls, and a button to control the volume.\\r\\n\\r\\n\\r\\nWe are offering a special discount for the first 100 customers to purchase the CO-1T. So, if you are looking for a great wireless headphone product with great battery life, then the CO-1T is the perfect choice for you.\\r\\n\\r\\n\\r\\nDon't miss out on this opportunity to try out our new product. Order now and experience the convenience of wireless headphones.\\r\\n\\r\\n\\r\\nIf you have any questions about the CO-1T, please feel free to contact us through our live chat support. We are always happy to help.\\r\\n\\r\\n\\r\\nThank you,\\r\\n[Your Name]\\r\\n\\r\\n\\r\\nIn this example, we changed the temperature value to 0.5. And throughout this chapter, you\\u2019ll see different temperature values being used in different situations. Increasing the temperature value tells the model to generate less predictable responses and instead be more \\u201ccreative.\\u201d And since this task does require some form of creativity (expanding brief keywords into an entire passage), we increased the temperature value. There is no one right value for a use case, so this is a setting you should try and iterate to get the best outcome for your task.\\r\\nWe have looked at one example, but writing as a use case can mean many different things. So instead of beginning your prompt with \\u201cCreate,\\u201d you can try other ways to extract different forms of content and formats that your application needs, such as using \\u201cExplain,\\u201d \\u201cGenerate,\\u201d \\u201cWrite,\\u201d \\u201cProvide,\\u201d \\u201cDesign,\\u201d \\u201cDescribe,\\u201d and so on.\\r\\nQuestion Answering\\r\\n!https://files.readme.io/f48f2db-Question_Answering.png\\r\\nThis use case is about answering a user's question. This could be a single-turn, question-answering scenario or a multi-turn, chatbot setting.\\r\\nQuestion answering can take place in either a closed or open setting. In a closed-book question-answering setting, we rely on the model to answer questions based on the general knowledge from which it has been trained. Here\\u2019s one example:\\r\\nPython\\r\\n`user_input =\\\"What features should I consider when choosing a wireless headphone\\\" prompt = user_input\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nSample response:\\r\\nThere are a few features to consider when choosing a wireless headphone. These include:\\r\\n\\r\\n\\r\\n1. Sound quality: It is important to choose a headphone that has good sound quality. This will ensure that you enjoy your music or podcasts.\\r\\n2. Comfort: It is important to choose a headphone that is comfortable to wear. This will ensure that you can wear it for long periods of time without experiencing any discomfort.\\r\\n3. Battery life: It is important to choose a headphone that has a long battery life. This will ensure that you can use it for long periods of time without having to recharge it.\\r\\n4. Price: It is important to choose a headphone that is within your budget. There are a variety of options available at different price points.\\r\\n\\r\\n\\r\\nThis works well for a question that is open-ended and doesn\\u2019t have one correct answer. But let\\u2019s say we have a customer asking for specific information about the wireless headphone, such as its feature set. For this, we need to turn to an extractive question-answering setting.\\r\\nIn this setting, we can get the model to refer to specific knowledge bases to help it do its job well. This way, we can design a system that can handle questions that require factual responses.\\r\\nHere, a customer asks a product question. We can append the customer\\u2019s question and the product description to the prompt, as follows.\\r\\nPython\\r\\n`user_input =\\\"How do I control the sound levels\\\"\\r\\nprompt = f\\\"\\\"\\\"{product} Given the product description above, answer this question: {user_input}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nAnd the model picks up the relevant information from the prompt to provide the correct answer.\\r\\nSample response:\\r\\nThe sound levels can be controlled by using the button on the headphones.\\r\\n\\r\\n\\r\\nBrainstorming\\r\\n!https://files.readme.io/ba6c262-Brainstorming.png\\r\\nAnother form of writing is brainstorming, where we want the model to generate a list of options based on a given prompt. This can be for writing outlines, generating ideas, providing critical analysis, and so on. This use case forces the model to go broad and cover different perspectives of a situation.\\r\\nIn this example, we want the model to act as an assistant to a customer support agent in identifying possible ways to troubleshoot a technical problem that a customer is facing.\\r\\nPython\\r\\n`user_input = \\\"I can't get the Bluetooth connection working\\\" prompt = f\\\"\\\"\\\"{product} A customer provided the following complaint about this product: {user_input}. Provide a bulleted list of possible ways to troubleshoot so we can advise the customer accordingly. \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.9) print(response)`\\r\\nSample response:\\r\\n- Check to see if the headphones are fully charged\\r\\n- Try resetting the headphones by holding the power button for 5 seconds\\r\\n- Make sure that your device is within range of the headphones\\r\\n- Try restarting your device\\r\\n- Check to see if the headphones are properly paired with your device\\r\\n- If you are still having trouble connecting, please contact customer service for further assistance.\\r\\n\\r\\n\\r\\nTransforming\\r\\n!https://files.readme.io/f1274c6-Transforming.png\\r\\nThe first thing that comes to mind when thinking about generative models is their ability to write a fresh piece of text, but one rather understated aspect is their ability to synthesize an existing piece of text.\\r\\nOne example is transforming a passage of text into a different form, making it reusable for different purposes.\\r\\nFor example, creating a list of Frequently Asked Questions (FAQs) about wireless headphones is crucial but requires some effort. We can cut short this process by getting the model to generate a list of FAQs based on the product description, as follows:\\r\\nPython\\r\\n`prompt =f\\\"\\\"\\\"Turn the following product description into a list of frequently asked questions (FAQ).\\r\\nProduct description: {product} \\\"\\\"\\\" response = generate_text(prompt, temp=0) print(response)`\\r\\nSample response:\\r\\nFrequently Asked Questions (FAQ)\\r\\n\\r\\n\\r\\nWhat is the CO-1T?\\r\\nThe CO-1T is a wireless headphone product that uses Bluetooth technology to connect to your devices. It has a long battery life and can be quickly charged using the included USB cable. The headphone is lightweight and comfortable, ideal for long periods of use. It has a built-in microphone for making calls, and a button to control the volume. The CO-1T is a great choice for anyone looking for a wireless headphone product with great battery life.\\r\\n\\r\\n\\r\\nHow long does the battery last?\\r\\nThe battery life of the CO-1T is long, and it can be quickly charged using the included USB cable.\\r\\n\\r\\n\\r\\nIs the CO-1T comfortable to wear?\\r\\nThe CO-1T is designed to be lightweight and comfortable, ideal for long periods of use.\\r\\n\\r\\n\\r\\nDoes the CO-1T have a built-in microphone?\\r\\nYes, the CO-1T has a built-in microphone for making calls.\\r\\n\\r\\n\\r\\nDoes the CO-1T have a button to control the volume?\\r\\nYes, the CO-1T has a button to control the volume.\\r\\n\\r\\n\\r\\nSummarizing\\r\\n!https://files.readme.io/d3e9e0c-Summarizing.png\\r\\nOne popular use case for synthesizing text is summarization. Here, we take a long passage of text and summarize it to its essence. These can be articles, conversation transcripts, reports, meeting notes, and so on.\\r\\nIn this example, we create a prompt to summarize a list of customer reviews about the wireless headphone.\\r\\nPython\\r\\n`user_input =\\\"\\\"\\\"Customer reviews of the CO-1T wireless headphones:\\r\\n\\\"The CO-1T is a great pair of headphones! The design is sleek and modern, and the headphones are \\\\ very comfortable to wear. The sound quality is excellent, and I can hear every detail of my music. \\\\ The built-in microphone means I can make calls without having to take my phone out of my pocket. I \\\\ highly recommend the CO-1T to anyone looking for a great pair of wireless headphones!\\\"\\r\\n\\\"I'm very disappointed with the CO-1T. The design is nice, but the battery life is terrible. I can \\\\ only use them for a few hours before they need to be recharged. This is very inconvenient, and I'm \\\\ not sure if I can recommend them to anyone.\\\"\\r\\n\\\"The CO-1T is a mixed bag. The speaker quality is great, but the built-in microphone's quality is \\\\ poor. I can hear every detail of my music, but my voice sounds distorted when I make calls. The \\\\ design is nice, and the headphones are comfortable to wear, but the battery life is only average. \\\\ I'm not sure if I can recommend them to anyone.\\\" \\\"\\\"\\\"\\r\\nprompt = f\\\"\\\"\\\"Summarize the following.\\r\\n{user_input} \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nExample response:\\r\\nThe CO-1T wireless headphones have mixed reviews. Some people love the design and sound quality, while others are disappointed with the battery life and microphone quality.\\r\\n\\r\\n\\r\\nAlternatively, there is a more streamlined way to generate quality summaries, and that is via the Summarize endpoint . It builds upon the Command model and is designed specifically to perform summarization. It supports a much longer context length than the Generate endpoint at 100,000 characters maximum. It also provides a much easier way to define the types of summaries, such as long vs. short, or paragraphs vs. bullets, without having to define the prompts.\\r\\nHere\\u2019s how to call the Summarize endpoint to summarize the customer reviews:\\r\\nPython\\r\\nresponse = co.summarize( text=user_input, length='short', format='paragraph', model='summarize-xlarge', temperature=0.3, additional_command=\\\"analyzing these customer reviews\\\" ) print(response.summary)\\r\\nSample response:\\r\\nPython\\r\\nThe CO-1T wireless headphones have received mixed reviews from customers. Some people love the design and sound quality, while others are disappointed with the battery life and microphone quality.\\r\\nRewriting\\r\\n!https://files.readme.io/5c4986a-Rewriting.png\\r\\nRewriting text is another useful use case where you need to modify some aspects of the text while maintaining its overall meaning. One example is changing the tone of a piece of text to tailor it to a specific audience. Here we want to rewrite the product description so it\\u2019s more relatable to students.\\r\\nPython\\r\\n`user_input = \\\"college students\\\"\\r\\nprompt = f\\\"\\\"\\\"Create a version of this product description that's tailored towards {user_input}.\\r\\n{product}\\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0.5) print(response)`\\r\\nSample response:\\r\\nThe CO-1T is a great wireless headphone option for college students who are always on the go. It uses Bluetooth technology to connect to your devices, so you don't have to worry about getting tangled up in cords. The battery life is long, and it can be quickly charged using the included USB cable. The headphone is lightweight and comfortable, making it ideal for long periods of use. It has a built-in microphone for making calls, and a button to control the volume. Plus, the price is just right for a college student's budget.\\r\\n\\r\\n\\r\\nAnother extremely useful way of looking at text synthesis is information extraction. Here, we leverage the model\\u2019s ability to capture the context of a piece of text to extract the correct information as specified by the prompt.\\r\\nHere is an example of an email that a customer is, unfortunately, asking for a refund for the wireless headphones. We can have the model process this email by getting it to extract information, such as the product name, refund reason, and pick-up address.\\r\\nPython\\r\\n`user_input =\\\"\\\"\\\"I am writing to request a refund for a recent CO-1T purchase I made on your platform. \\\\ Unfortunately, the produce has not met my expectations due to its poor battery life. \\\\ Please arrange for the pick-up at this address: to 171 John Street, Toronto ON, M5T 1X2.\\\"\\\"\\\"\\r\\nprompt =f\\\"\\\"\\\"Extract the product, refund reason and pick-up address from this email:\\r\\n{user_input} \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nSample response:\\r\\nProduct: CO-1T\\r\\nRefund reason: Poor battery life\\r\\nPick-up address:  171 John Street, Toronto ON, M5T 1X2\\r\\n\\r\\n\\r\\nClassifying\\r\\n!https://files.readme.io/49c3495-Classifying.png\\r\\nOne of the most widely deployed use cases in NLP is text classification. Here, the task is to classify a piece of text into one of a few predefined classes. In this example, we want to classify incoming customer messages into one of three categories: Order, Support, or Refunds.\\r\\nWe can create the prompt as follows.\\r\\nPython\\r\\n`user_input =\\\"\\\"\\\"The battery drains fast\\\"\\\"\\\"\\r\\nprompt = f\\\"\\\"\\\"The following is a user message to a customer support agent. Classify the message into one of the following categories: Order, Support, or Refunds.\\r\\n{user_input} \\\"\\\"\\\"\\r\\nresponse = generate_text(prompt, temp=0) print(response)`\\r\\nHere\\u2019s a sample response where the generative model correctly classifies the text into the right category:\\r\\nSupport\\r\\n\\r\\n\\r\\nAlternatively, the Classify endpoint provides a simple API for running text classification. The endpoint leverages Cohere\\u2019s embeddings models and makes it easy to add training examples and even create custom models that are specifically tailored to your task.\\r\\nHere\\u2019s how we can use the Classify endpoint. It requires a minimum of two examples per class, which is passed as an argument to the API call. We have six examples altogether \\u2013 two for each class.\\r\\nPython\\r\\n`from cohere.responses.classify import Example\\r\\nresponse = co.classify( model='embed-english-v3.0', inputs=[user_input], examples=[Example(\\\"I can\\\\'t connect to the bluetooth\\\", \\\"Support\\\"), Example(\\\"Why is max volume so low\\\", \\\"Support\\\"), Example(\\\"When will my order arrive\\\", \\\"Order\\\"), Example(\\\"How much is the shipping cost\\\", \\\"Order\\\"), Example(\\\"What is your refund policy\\\", \\\"Refunds\\\"), Example(\\\"How do I return my product\\\", \\\"Refunds\\\")])\\r\\nprint(response.classifications[0].prediction)`\\r\\nSample response:\\r\\nSupport\\r\\n\\r\\n\\r\\nConclusion\\r\\nThis chapter provides a starting point for understanding the range of use cases you can build with the Cohere Command model. Take these use cases as the initial set of examples to further experiment on what\\u2019s possible.\",\n",
            "      \"title\": \"Use Case Patterns\",\n",
            "      \"url\": \"https://docs.google.com/document/d/1X9XM5nLjjF_8b6ulj35FK8ioWCJTg6n0KtYtXLkr9Dw/edit?usp=drivesdk\"\n",
            "    },\n",
            "    {\n",
            "      \"editedBy\": \"Meor Amer\",\n",
            "      \"id\": \"1wngAfCJY1IgD6H__4AkQXFfymKUpSeJL13TItbigdyA\",\n",
            "      \"mimeType\": \"application/vnd.google-apps.document\",\n",
            "      \"modifiedTime\": \"2023-12-01T07:50:02.204Z\",\n",
            "      \"text\": \"\\ufeffValidating Outputs\\r\\nIn this chapter, you'll learn how to implement validation on LLM outputs.\\r\\nSuggest Edits\\r\\n!https://files.readme.io/1a4c229-validating-llm-outputs.png\\r\\nThis chapter was written in collaboration with the Guardrails AI team.\\r\\nIntroduction\\r\\nIn previous chapters, we covered prompt engineering techniques to elicit the desired responses from a large language model (LLM).\\r\\nHowever, one key property of LLMs that\\u2019s different from traditional software is that the output is probabilistic in nature. The same input (i.e., the prompt) may not always produce the same response. While this property makes it possible to build entirely new classes of natural language applications, it also means that those applications require a mechanism for validating their outputs.\\r\\nHere\\u2019s an example. In Chapter 1 , we looked at a text extraction task of turning a list of bank invoices in a text document into a JSON object containing three fields: \\u201cInvoice Number,\\u201d \\u201cMerchant Name,\\u201d and \\u201cAccount Number.\\u201d For brevity, we\\u2019ll turn it into a shorter version with the document containing just one invoice, as follows.\\r\\nPython\\r\\nprompt=\\\"\\\"\\\"Turn the following information into a JSON string with the following keys: Invoice Number, Merchant Name, and Account Number. Bank Invoice: INVOICE #0521 MERCHANT ALLBIRDS ACC XXX3846 \\\"\\\"\\\"\\r\\nThis produced an LLM response that followed exactly what we wanted, as shown below.\\r\\nJSON\\r\\n{ \\\"Invoice Number\\\": \\\"0521\\\", \\\"Merchant Name\\\": \\\"Allbirds\\\", \\\"Account Number\\\": \\\"XXXX3846\\\" }\\r\\nBut how do we ensure we\\u2019ll get the same response every time? Perhaps another time, the output may miss some information, such as the returning incomplete information like this one.\\r\\nJSON\\r\\n{ \\\"Invoice Number\\\": \\\"0521\\\" }\\r\\nThere are many other ways that a response may not match the expected structure. For example, what if the generated output contained fields we never specified? What if the document provided were more challenging to parse, causing an unclear response? What if we wanted to impose a rule that a field can only take up values within a specific range? In these cases and many others, we must add a step to validate the output.\\r\\nIn the rest of this chapter, we\\u2019ll look at LLM output validation and examples of how to implement it using an open-source package called Guardrails AI. This chapter comes with a Google Colaboratory notebook .\\r\\nWhen Is Output Validation Needed\\r\\nTo build robust and production-ready LLM applications, the outputs need to be predictable. This helps to safeguard the application against unexpected behaviors and to ensure a reliable user experience. The following provides some example scenarios when output validation may be needed.\\r\\n* Structure compliance : Some LLM applications require their output to go beyond just freeform text and instead follow a specific structure containing specific types of information. We looked at a toy example earlier of a text extraction task that requires a JSON output that follows a certain format. We can extend this to synthetic data generation cases, where the generated data must meet certain criteria.\\r\\n* Safe responses : Due to their probabilistic nature, LLM applications require additional guardrails to ensure their outputs are safe, ethical, and privacy-preserving. For example, we may want to confirm that an output does not contain profanity. Alternatively, we may want to ensure that an output does not contain personally identifiable information (PII).\\r\\n* Semantic similarity : In some applications, we may require the output to be similar enough to a target. For example, in text summarization tasks, we want to ensure that the summary does not deviate too far from the original document. For this, with the help of text embeddings, we want to validate that the summary and the document are similar enough semantically.\\r\\n* Valid choices : We may also want the LLM to generate valid outputs per given definitions or constraints. For example, creating an LLM chess player will require the LLM output to generate only valid moves for a given board state. We could also validate that a generated piece of code is syntactically correct.\\r\\n* Quality assurance : More generally, we may want to implement a validation step to ensure an LLM output meets a certain quality standard for a respective application\\u2019s use case and provides value to users.\\r\\nWe can bake these types of validation into an application by implementing a set of validation steps followed by an automated action to fix any gaps found. Let\\u2019s see how we can do this.\\r\\nIntroduction to Guardrails AI\\r\\nGuardrails AI is a Python package that enables developers to enhance the outputs of LLms by adding structural, type, and quality assurances. Guardrails helps developers with two key activities in a validation process:\\r\\n* Validation : Performing output validation for LLMs in a similar style to the Pydantic package, which is a data validation library for Python. Some examples are validating the structure of generated JSON output, identifying bias in generated text, and checking for bugs in generated code.\\r\\n* Correction : Performing corrective actions based on the validation results, such as asking the LLM to re-generate the response or fixing the output directly.\\r\\n!https://txt.cohere.com/content/images/2023/09/Output-Validation.png\\r\\nValidation and correction are two key activities in a validation process\\r\\nImplementation-wise, the following steps are involved in incorporating Guardrails into an LLM application:\\r\\n* Create the output schema : The RAIL spec contains the overall schema of the LLM output, the type info for each field, the quality criteria for each field, and the corrective action to be taken if the quality criteria are not met. It also contains the prompt template and any custom code for implementing the schema. Alternatively, the output schema can also be defined using Pydantic, which we\\u2019ll use in our example later.\\r\\n* Initialize a Guard object based on the schema : The Guard class is the main entry point for using Guardrails. It is initialized using the output schema created in the previous step.\\r\\n* Wrap an LLM call with the Guard object : The gd.Guard object wraps around an LLM API call to validate, structure, and correct the outputs.\\r\\n!https://txt.cohere.com/content/images/2023/09/Output-Validation-with-Guardrails-AI.png\\r\\nThe key steps in output validation with Guardrails: creating the output schema, initializing a Guard object, and wrapping an LLM call with it\\r\\nLet\\u2019s look at an example of using Guardrails in a text extraction task. The task is to extract the information from a doctor\\u2019s note into a JSON object. The following is the doctor\\u2019s note.\\r\\nPython\\r\\ndoctors_notes = \\\"\\\"\\\"49 y/o Male with chronic macular rash to face & hair, worse in beard, eyebrows & nares. Itchy, flaky, slightly scaly. Moderate response to OTC steroid cream\\\"\\\"\\\"\\r\\nSpecifically, we want our extracted information to contain the following fields:\\r\\n* Patient's gender\\r\\n* Patient's age\\r\\n* A list of symptoms, each with a severity rating and an affected area\\r\\n* A list of medications, each with information about the patient's response to the medication\\r\\nSetup\\r\\nFirst, let\\u2019s install the packages required: cohere , guardrails-ai , and pydantic .\\r\\npip install \"cohere<5\" guardrails-ai pydantic -q\\r\\n\\r\\n\\r\\nNext, import the packages.\\r\\nPython\\r\\nimport cohere import guardrails as gd from guardrails.validators import ValidRange, ValidChoices from pydantic import BaseModel, Field from rich import print from typing import List\\r\\nDefine the Output Schema\\r\\nNext, we define the output schema that defines what the LLM response should look like. As mentioned earlier, Guardrails provides an option to define the schema using Pydantic. We\\u2019ll use this option, and below is the schema we\\u2019ll use for the doctor notes extraction task.\\r\\nPython\\r\\n`class Symptoms(BaseModel): symptom: str = Field(..., description=\\\"Symptom that a patient is experiencing\\\") affected_area: str = Field( ..., description=\\\"What part of the body the symptom is affecting\\\", validators=[ValidChoices([\\\"head\\\", \\\"neck\\\", \\\"chest\\\"], on_fail=\\\"reask\\\")] )\\r\\nclass CurrentMeds(BaseModel): medication: str = Field(..., description=\\\"Name of the medication the patient is taking\\\") response: str = Field(..., description=\\\"How the patient is responding to the medication\\\")\\r\\nclass PatientInfo(BaseModel): gender: str = Field(..., description=\\\"Patient's gender\\\") age: int = Field(..., description=\\\"Patient's age\\\", validators=[ValidRange(0, 100)]) symptoms: List[Symptoms] = Field(..., description=\\\"Symptoms that the patient is experiencing\\\") current_meds: List[CurrentMeds] = Field(..., description=\\\"Medications that the patient is currently taking\\\")`\\r\\nIn the schema, we defined a few \\u201cvalidators,\\u201d a Guardrails feature that lets us define the type of validation to perform. One example is ValidChoices , useful for situations where we want to enforce that a response can only be within a predefined list of items. In our example, in the symptom field, the value can only be one of head, neck, or chest. And if the generated response doesn\\u2019t fulfill this criteria, it will be re-prompted. This is shown by the on_fail setting that triggers a reask .\\r\\nThe Guardrails documentation provides more information about the types of validators in Guardrails.\\r\\nInitialize a Guard Object Based on the Schema\\r\\nNext, we initialize a Guard object based on the schema we have defined.\\r\\nFirst, we define the base instruction prompt for the LLM as follows.\\r\\nPython\\r\\n`PROMPT = \\\"\\\"\\\"Given the following doctor's notes about a patient, please extract a dictionary that contains the patient's information.\\r\\n{{doctors_notes}}\\r\\n@complete_json_suffix_v2 \\\"\\\"\\\"`\\r\\nThen, we initialize a Guard object from the PatientInfo Pydantic model.\\r\\nPython\\r\\nguard = gd.Guard.from_pydantic(PatientInfo, prompt=PROMPT) print(guard.base_prompt)\\r\\nGuardrails then uses this information to construct the full prompt for the LLM, which looks like the following example.\\r\\nGiven the following doctor's notes about a patient,\\r\\nplease extract a dictionary that contains the patient's information.\\r\\n\\r\\n\\r\\n{doctors_notes}\\r\\n\\r\\n\\r\\nGiven below is XML that describes the information to extract from this document and the tags to extract it into.\\r\\n\\r\\n\\r\\n<output>\\r\\n    <string name=\\\"gender\\\" description=\\\"Patient's gender\\\"/>\\r\\n    <integer name=\\\"age\\\" format=\\\"valid-range: min=0 max=100\\\" description=\\\"Patient's age\\\"/>\\r\\n    <list name=\\\"symptoms\\\" description=\\\"Symptoms that the patient is experiencing\\\">\\r\\n        <object>\\r\\n            <string name=\\\"symptom\\\" description=\\\"Symptom that a patient is experiencing\\\"/>\\r\\n            <string name=\\\"affected_area\\\" format=\\\"valid-choices: choices=['head', 'neck', 'chest']\\\"\\r\\ndescription=\\\"What part of the body the symptom is affecting\\\"/>\\r\\n        </object>\\r\\n    </list>\\r\\n    <list name=\\\"current_meds\\\" description=\\\"Medications that the patient is currently taking\\\">\\r\\n        <object>\\r\\n            <string name=\\\"medication\\\" description=\\\"Name of the medication the patient is taking\\\"/>\\r\\n            <string name=\\\"response\\\" description=\\\"How the patient is responding to the medication\\\"/>\\r\\n        </object>\\r\\n    </list>\\r\\n</output>\\r\\n\\r\\n\\r\\nONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name`\\r\\nattribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON\\r\\nMUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and\\r\\nspecific types. Be correct and concise.\\r\\n\\r\\n\\r\\nHere are examples of simple (XML, JSON) pairs that show the expected behavior:\\r\\n- `<string name='foo' format='two-words lower-case' />` => `{{'foo': 'example one'}}`\\r\\n- `<list name='bar'><string format='upper-case' /></list>` => `{{\\\"bar\\\": ['STRING ONE', 'STRING TWO', etc.]}}`\\r\\n- `<object name='baz'><string name=\\\"foo\\\" format=\\\"capitalize two-words\\\" /><integer name=\\\"index\\\" format=\\\"1-indexed\\\"\\r\\n/></object>` => `{{'baz': {{'foo': 'Some String', 'index': 1}}}}`\\r\\n\\r\\n\\r\\nWrap an LLM Call with the Guard Object\\r\\nWe\\u2019re ready to run an LLM call using the Cohere Generate endpoint. For this, we wrap the LLM call with the Guard object. This means it will take care of the validation and reasking (if any) until the final generated output fulfills the defined schema.\\r\\nPython\\r\\nco = cohere.Client(api_key='COHERE_API_KEY') raw_llm_output, validated_output = guard( co.generate, prompt_params={\\\"doctors_notes\\\": doctors_notes}, model='command', max_tokens=1024, temperature=0.6 )\\r\\nAnd we get the final validated output as follows.\\r\\nJSON\\r\\n{ 'patient_info': { 'gender': 'Male', 'age': 49, 'symptoms': [ {'symptom': 'Chronic macular rash', 'affected area': 'head'}, { 'symptom': 'Itchy, flaky, slightly scaly', 'affected area': 'head' } ], 'current_meds': [ {'medication': 'OTC steroid cream', 'response': 'Moderate'} ] } }\\r\\nBehind the scenes, Guardrails performs the validation step on the output against the schema, raises any errors if there are mismatches, and triggers a reask. We can trace the execution steps as follows.\\r\\nPython\\r\\nprint(guard.state.most_recent_call.history[0].rich_group)\\r\\nThe LLM call first returned the following response. Two symptoms were generated, and for both \\u201cFace & hair\\u201d and \\u201cBeard, eyebrows & nares,\\u201d the affected areas did not fall within the three options we had defined earlier.\\r\\nJSON\\r\\n{ \\\"patient_info\\\": { \\\"gender\\\": \\\"Male\\\", \\\"age\\\": 49, \\\"symptoms\\\": [ { \\\"symptom\\\": \\\"Chronic macular rash\\\", \\\"affected area\\\": \\\"Face & hair\\\" }, { \\\"symptom\\\": \\\"Itchy, flaky, slightly scaly\\\", \\\"affected area\\\": \\\"Beard, eyebrows & nares\\\" } ], \\\"current_meds\\\": [ { \\\"medication\\\": \\\"OTC steroid cream\\\", \\\"response\\\": \\\"Moderate\\\" } ] } }\\r\\nGuardrails captured this discrepancy by raising a FieldReAsk object containing the incorrect value, the error message, and other additional information.\\r\\nJSON\\r\\n{ 'patient_info': { 'gender': 'Male', 'age': 49, 'symptoms': [ { 'symptom': 'Chronic macular rash', 'affected area': FieldReAsk( incorrect_value='Face & hair', error_message=\\\"Value Face & hair is not in choices ['head', 'neck', 'chest'].\\\", fix_value=None, path=['patient_info', 'symptoms', 0, 'affected area'] ) }, { 'symptom': 'Itchy, flaky, slightly scaly', 'affected area': FieldReAsk( incorrect_value='Beard, eyebrows & nares', error_message=\\\"Value Beard, eyebrows & nares is not in choices ['head', 'neck', 'chest'].\\\", fix_value=None, path=['patient_info', 'symptoms', 1, 'affected area'] ) } ], 'current_meds': [ {'medication': 'OTC steroid cream', 'response': 'Moderate'} ] } }\\r\\nBased on this information, it triggered another LLM call to re-generate the response. Here is the full prompt.\\r\\nI was given the following JSON response, which had problems due to incorrect values.\\r\\n\\r\\n\\r\\n {\\r\\n   \\\"patient_info\\\": {\\r\\n     \\\"symptoms\\\": [\\r\\n       {\\r\\n         \\\"affected area\\\": {\\r\\n           \\\"incorrect_value\\\": \\\"Face & hair\\\",\\r\\n           \\\"error_message\\\": \\\"Value Face & hair is not in choices ['head', 'neck', 'chest'].\\\"\\r\\n        }\\r\\n       },\\r\\n       {\\r\\n         \\\"affected area\\\": {\\r\\n           \\\"incorrect_value\\\": \\\"Beard, eyebrows & nares\\\",\\r\\n           \\\"error_message\\\": \\\"Value Beard, eyebrows & nares is not in choices ['head', 'neck', 'chest'].\\\"\\r\\n         }\\r\\n       }\\r\\n     ]\\r\\n   }\\r\\n }\\r\\n\\r\\n\\r\\n Help me correct the incorrect values based on the given error messages.\\r\\n\\r\\n\\r\\n Given below is XML that describes the information to extract from this document and the tags to extract it\\r\\n into.\\r\\n\\r\\n\\r\\n <output>\\r\\n     <object name=\\\"patient_info\\\">\\r\\n         <list name=\\\"symptoms\\\" description=\\\"Symptoms that the patient is currently experiencing. Each symptom\\r\\n should be classified into a separate item in the list.\\\">\\r\\n             <object>\\r\\n                 <string name=\\\"affected area\\\" description=\\\"What part of the body the symptom is affecting\\\"\\r\\n format=\\\"lower-case; valid-choices: choices=['head', 'neck', 'chest']\\\"/>\\r\\n             </object>\\r\\n         </list>\\r\\n     </object>\\r\\n </output>\\r\\n\\r\\n\\r\\n ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name`\\r\\n attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The\\r\\n JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects\\r\\n and specific types. Be correct and concise. If you are unsure anywhere, enter `null`.\\r\\n\\r\\n\\r\\nThis returned the following response.\\r\\nJSON\\r\\n{ \\\"patient_info\\\": { \\\"symptoms\\\": [ { \\\"affected area\\\": \\\"head\\\", \\\"error_message\\\": null }, { \\\"affected area\\\": \\\"head\\\", \\\"error_message\\\": null } ] } }\\r\\nGuardrails then generated the final validated output, which now completely fulfills the schema.\\r\\nJSON\\r\\n{ 'patient_info': { 'gender': 'Male', 'age': 49, 'symptoms': [ {'symptom': 'Chronic macular rash', 'affected area': 'head'}, { 'symptom': 'Itchy, flaky, slightly scaly', 'affected area': 'head' } ], 'current_meds': [ {'medication': 'OTC steroid cream', 'response': 'Moderate'} ] } }\\r\\nConclusion\\r\\nIn this chapter, we looked at LLM output validation and how to implement it using Guardrails AI. Output validation is key to ensuring a generative AI application is robust and predictable enough to be deployed confidently.\",\n",
            "      \"title\": \"Validating Outputs\",\n",
            "      \"url\": \"https://docs.google.com/document/d/1wngAfCJY1IgD6H__4AkQXFfymKUpSeJL13TItbigdyA/edit?usp=drivesdk\"\n",
            "    },\n",
            "    {\n",
            "      \"editedBy\": \"Meor Amer\",\n",
            "      \"id\": \"10x9mJOnEr62hg1IFxgAtD1aIFS4NXJ2l5Lt-UhJXLVg\",\n",
            "      \"mimeType\": \"application/vnd.google-apps.document\",\n",
            "      \"modifiedTime\": \"2024-02-29T06:01:59.503Z\",\n",
            "      \"text\": \"\\ufeffEvaluating Outputs\\r\\nIn this chapter, you'll learn about the different techniques for evaluating LLM outputs.\\r\\nSuggest Edits\\r\\n!https://files.readme.io/7eb0059-Evaluating_LLM_Outputs.png\\r\\nIntroduction\\r\\nLarge language models (LLMs) offer exciting new ways to build applications that leverage natural language as the interface. However, as impressive as this technology may be, it is crucial to evaluate the generated outputs of LLMs to ensure the quality of an application.\\r\\nEvaluating LLM outputs is especially important because the outputs produced by these models are probabilistic \\u2013 meaning the same prompt does not necessarily produce the same outputs every time. Evaluations provide a way to measure the quality level of the outputs, ensuring a great user experience.\\r\\nIn this chapter, we look at a few techniques for evaluating the outputs generated by an LLM.\\r\\nReal User Feedback\\r\\nThe gold standard for evaluation is gathering actual feedback from real application users. The best way to gain insights into an application\\u2019s quality and usefulness is by collecting feedback from users who interact with the application. In contrast, the rest of the methods we\\u2019ll discuss are all proxies for understanding real user experience and behaviors.\\r\\nThe specific tactics for gathering user feedback can come in different forms, for example:\\r\\n* Explicit feedback : By implementing features to gather user feedback, such as thumbs up/down for an output, rating the output, and more\\r\\n* Implicit feedback : By observing user behaviors, such as considering ignored outputs as negatives, analyzing time spent on the output, and more\\r\\nThis creates a flywheel for continuously improving an application. As more users start using the application, more data becomes available to inform the effectiveness of an application, providing signals on areas for improvement.\\r\\nBut, of course, the challenge is that this can happen only after we deploy the application and users have been using it for a while. So, we must also perform evaluations before an application is deployed. For this, let\\u2019s look at alternative evaluation methods.\\r\\nHuman Evaluation\\r\\nThe next best option is to have human annotators evaluate the outputs of an application in the pre-deployment stage. A typical evaluation approach requires building a test dataset, and evaluation is performed against this test dataset.\\r\\n!https://txt.cohere.com/content/images/2023/10/Human-evaluation.png\\r\\nAn example human evaluation comparing two LLM outputs.\\r\\nLet\\u2019s look at a question-answering example. Here, we have one test data point: the prompt asks a question about a headphone product given a user review. The evaluation task is to rate the response generated by the LLM.\\r\\nGiven the following user review, answer the question.\\r\\n\\r\\n\\r\\nUser review: The CO-1T is a great pair of headphones! The sound quality is the best out there, and I can hear every detail of my music.\\r\\n\\r\\n\\r\\nQuestion: Why is the CO-1T a great wireless headphone?\\r\\n\\r\\n\\r\\nAnd let\\u2019s suppose that this is the generated response.\\r\\n\\\"Because the audio experience is unrivaled\\\"\\r\\n\\r\\n\\r\\nThere are several ways to approach evaluation for this response, as follows.\\r\\nReference\\r\\nHere, the evaluator would compare each test data point against the ground truth of an ideal response.\\r\\nWith our example, the ideal response might be, \\u201cBecause the sound quality is the best out there.\\u201d Based on this, the evaluator provides a Yes/No judgment on whether the generated response provides an accurate response.\\r\\nBut this approach requires the ground truth to be constructed beforehand. Given that no two use cases are quite the same, this means having to construct ground truth for each prompt and application. Moreover, the quality of the ground truth directly affects the evaluation outcome \\u2013 if not constructed correctly, it can produce misleading results.\\r\\nScoring\\r\\nHere, the evaluator would evaluate the generated response by assigning a score, such as a rating between 0 and 10. There is no ground truth as a reference, so it\\u2019s up to the evaluator to provide a verdict on the quality of an output.\\r\\nThe score can be a single score or a set of scores, and it can be broad or granular, depending on the use case. For example, a creative writing task might require more granular scoring for different output characteristics, such as fluency, interestingness, and conciseness.\\r\\nAn evaluation criterion can be a score along a scale, and it can also be a check against a flag. For example, a summarization task might require checking whether the output is consistent and does not generate content that doesn't exist in the actual document. A specialized task might require checking for specific rules, such as the number of syllables in a haiku.\\r\\nA/B Testing\\r\\nHere, the evaluator would be given a pair of LLM-generated responses and asked to rate the better response. This is useful for comparing an application\\u2019s quality over different time snapshots or different sets of configurations, such as prompts and parameters.\\r\\nWith our example, let\\u2019s suppose the evaluator is tasked to compare the following two responses to our user question above.\\r\\n1. \\\"Because the audio experience is unrivaled\\\"\\r\\n2. \\\"Because the microphone has the best quality\\\"\\r\\n\\r\\n\\r\\nWe can probably agree that the winning response should be the first one (\\u201cBecause the audio experience is unrivaled\\\") as it answers the question accurately, while the second response (\\u201cBecause the microphone has the best quality\\u201d) talks about the microphone\\u2019s quality instead of the sound quality.\\r\\nThe challenge with human evaluation is that it cannot be scaled efficiently. The cost and time incurred are significantly higher than the alternative methods, which can be automated.\\r\\nAnother challenge is that human evaluation is subjective \\u2013 the verdict from one evaluator may not be the same as another. The example above is relatively straightforward, but in more challenging tasks, there will be more ambiguity and room for interpretation about what makes a response good or bad. Many factors can influence an evaluator\\u2019s verdict, such as expertise, style, and biases, impacting the evaluation outcome.\\r\\nLLM-Generated Evaluation\\r\\nAn alternative to human evaluation is to have an LLM to evaluate the output. With the same setup as above, this is done by having a prompt instructing the LLM to provide the verdict of a generated answer against the reference.\\r\\n!https://txt.cohere.com/content/images/2023/10/LLM-Generated-Evaluation.png\\r\\nAn example LLM-generated evaluation comparing two LLM outputs.\\r\\nAny human evaluation paradigms that we discussed (reference, scoring, and A/B testing) could be replicated using LLM-generated evaluation.\\r\\nThe example below uses the Command model to perform an A/B testing evaluation for the same question-answering task. The model's task is to choose the winning response between two responses to the question.\\r\\nPython\\r\\n`# Add text to evaluate ref_answer = \\\"Because the sound quality is the best out there\\\" gen_answers = [\\\"Because the audio experience is unrivaled\\\", \\\"Because the microphone has the best quality\\\"]\\r\\nRun evaluation\\r\\nfor gen_answer in gen_answers: prompt=f\\\"\\\"\\\"User review: The CO-1T is a great pair of headphones! The sound quality is the \\\\ best out there, and I can hear every detail of my music. Question: Why is the CO-1T a great headphone? Answer #1: {gen_answers[0]} Answer #2: {gen_answers[1]}\\r\\nGiven the question about the user review, state whether Answer #1 or Answer #2 provides \\\\\\\\\\r\\nthe more accurate answer.\\\"\\\"\\\"\\r\\n\\r\\n\\r\\nresponse = co.generate(prompt=prompt,max_tokens=50, temperature=0).generations[0].text\\r\\nprint(response)`\\r\\n\\r\\n\\r\\nThe generated verdict is \\u201cAnswer #1\\u201d, which is what we expect the winning response should be.\\r\\nAnswer #1\\r\\n\\r\\n\\r\\nThis approach is promising as it eliminates the cost and time constraints of human evaluation, but the jury is still out on whether it can surpass human evaluation in accuracy and quality. Its effectiveness on one task doesn\\u2019t guarantee that it will generalize to other tasks and domains, and the only way to know is by testing it on a specific application.\\r\\nLLM-generated evaluation also faces the same subjectivity challenge as human evaluation. Many factors can affect a model\\u2019s evaluation outcome, such as the model\\u2019s overall capability, whether it\\u2019s being trained specifically to perform evaluations, the presence of training data that could introduce biases, and more.\\r\\nWord-Level Metrics\\r\\nAnother evaluation approach compares the reference and generated output at the word/token (or word/token group) level. Several evaluation metrics are available, such as BLEU , ROUGE , Perplexity , and BERTScore .\\r\\n!https://txt.cohere.com/content/images/2023/10/Word-level-Metrics.png\\r\\nLet\\u2019s look at an example with ROUGE , originally created for evaluating summaries. It measures the number of matching \\u201cn-grams\\u201d between the reference and generated text. An N-gram is a contiguous sequence of n items in a text, where n can be 1, 2, and so on. To keep it simple, we\\u2019ll use n=1, also called \\u201cunigrams.\\u201d For example, in the sentence \\u201cI love cats,\\u201d the unigrams are \\u201cI,\\u201d \\u201clove,\\u201d and \\u201ccats.\\u201d\\r\\nWe calculate the precision, recall, and F1-score of the n-grams of the question-answering task against a reference answer of \\u201cBecause the sound quality is the best out there.\\u201d\\r\\n* Precision is the ratio of the count of matching unigrams divided by the count of unigrams in the generated text\\r\\n* Recall is the ratio of the count of matching unigrams divided by the count of unigrams in the reference text\\r\\n* F1-score is calculated from precision and recall with the following formula: 2 * (precision * recall) / (precision + recall)\\r\\nHere is an example using ROUGE:\\r\\nPython\\r\\n`from collections import Counter\\r\\ndef rouge_1(reference, candidate): # Turn into unigrams reference_words = reference.split() candidate_words = candidate.split()\\r\\n# Compute the number of overlapping words\\r\\nreference_count = Counter(reference_words)\\r\\ncandidate_count = Counter(candidate_words)\\r\\noverlap = sum(min(candidate_count[w], reference_count[w]) for w in candidate_count)\\r\\n\\r\\n\\r\\n# Compute precision, recall, and F1 score\\r\\nrecall = overlap / len(reference_words)\\r\\nprecision = overlap / len(candidate_words)\\r\\nf1 = 2 * (recall * precision) / (recall + precision)\\r\\n\\r\\n\\r\\n# Return resuls\\r\\nreturn {\\\"recall\\\": recall, \\\"precision\\\": precision, \\\"f1\\\": f1}\\r\\n\\r\\n\\r\\nfor idx,gen_answer in enumerate(gen_answers): result = rouge_1(ref_answer, gen_answer) print(f\\\"Answer #{idx+1}\\\") print(f\\\"Precision: {result['precision']:.2f}\\\") print(f\\\"Recall: {result['recall']:.2f}\\\") print(f\\\"F1-Score: {result['f1']:.2f}\\\") print(\\\"\\\\n\\\")`\\r\\nThis gives the following outcome.\\r\\nAnswer #1\\r\\nPrecision: 0.50\\r\\nRecall: 0.33\\r\\nF1-Score: 0.40\\r\\n\\r\\n\\r\\nAnswer #2\\r\\nPrecision: 0.71\\r\\nRecall: 0.56\\r\\nF1-Score: 0.63\\r\\n\\r\\n\\r\\nHere, the second generated answer scored higher than the first in precision, recall, and F1-score, which is not the expected outcome. This is because it has more unigram overlaps with the reference answer, for example, with the words \\u201cbest\\u201d and \\u201cquality.\\u201d\\r\\nThis is an example where word-level metrics may fall short. They can be handy because they are easy to interpret and their implementation is fast and cheap, but they may not capture the overall meaning and accuracy when comparing two pieces of text.\\r\\nConclusion\\r\\nThis chapter looked at a few techniques for evaluating LLM outputs, from human evaluations to automated ones.\\r\\nThere is a trade-off to be considered: on the one hand, automated evaluations are much more cost and time-efficient, which makes them practical options in some cases, such as in the early prototyping stages. On the other hand, human evaluations are still the gold standard for getting the strongest signal on an application's accuracy and usefulness.\\r\\nIt\\u2019s also worth noting that this chapter looks at evaluating LLM outputs in general without making any assumptions about the actual task. In practice, there are other approaches not mentioned here that better suit specific tasks, such as code execution or information extraction.\\r\\nUltimately, each evaluation approach has its potential pitfalls. An evaluation outcome can only be considered reliable if we have first understood and mitigated the associated limitations.\\r\\nOriginal Source\\r\\nThis material comes from the post: Evaluating LLM Outputs .\\r\\nUpdated about 1 month ago\\r\\n________________\\r\\n\\r\\n\\r\\nWhat\\u2019s Next\\r\\n* Conclusion - Prompt Engineering\\r\\n* Table of Contents\\r\\n* IntroductionReal User FeedbackHuman EvaluationReferenceScoringA/B TestingLLM-Generated EvaluationWord-Level MetricsConclusionOriginal Source\",\n",
            "      \"title\": \"Evaluating Outputs\",\n",
            "      \"url\": \"https://docs.google.com/document/d/10x9mJOnEr62hg1IFxgAtD1aIFS4NXJ2l5Lt-UhJXLVg/edit?usp=drivesdk\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "! curl --request POST \\\n",
        "\t\t--url https://demo-conn-gdrive.onrender.com/search \\\n",
        "\t\t--header 'Authorization: Bearer YOUR_CONNECTOR_API_KEY' \\\n",
        "\t\t--header 'Content-Type: application/json' \\\n",
        "\t\t--data '{\"query\": \"prompt chaining\"}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbRh6GNuRnmw"
      },
      "source": [
        "# Create a chatbot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0V7dWvznljbD"
      },
      "source": [
        "The `Chatbot` class below handles the interaction between the user and chatbot.  We define the connector for the chatbot to use with the attribute `self.connectors`.\n",
        "\n",
        "The run() method contains the logic for getting the user message, displaying the chatbot response with citations, along with a way for the user to end the conversation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "PB5kpanxRnmw",
        "outputId": "ee14bf68-0921-41c8-b6c3-bc652a107a4d"
      },
      "outputs": [],
      "source": [
        "class Chatbot:\n",
        "    def __init__(self, connectors: List[str]):\n",
        "        \"\"\"\n",
        "        Initializes an instance of the Chatbot class.\n",
        "\n",
        "        \"\"\"\n",
        "        self.conversation_id = str(uuid.uuid4())\n",
        "        self.connectors = [ChatConnector(id=connector) for connector in connectors]\n",
        "\n",
        "    def run(self):\n",
        "        \"\"\"\n",
        "        Runs the chatbot application.\n",
        "\n",
        "        \"\"\"\n",
        "        while True:\n",
        "            # Get the user message\n",
        "            message = input(\"User: \")\n",
        "\n",
        "            # Typing \"quit\" ends the conversation\n",
        "            if message.lower() == \"quit\":\n",
        "                print(\"Ending chat.\")\n",
        "                break\n",
        "            else:                       # If using Google Colab, remove this line to avoid printing the same thing twice\n",
        "              print(f\"User: {message}\") # If using Google Colab, remove this line to avoid printing the same thing twice\n",
        "\n",
        "            # Generate response\n",
        "            response = co.chat_stream(\n",
        "                    message=message,\n",
        "                    model=\"command-r-plus\",\n",
        "                    conversation_id=self.conversation_id,\n",
        "                    connectors=self.connectors,\n",
        "            )\n",
        "\n",
        "            # Print the chatbot response, citations, and documents\n",
        "            print(\"\\nChatbot:\")\n",
        "            citations = []\n",
        "            cited_documents = []\n",
        "\n",
        "            # Display response\n",
        "            for event in response:\n",
        "                if event.event_type == \"text-generation\":\n",
        "                    print(event.text, end=\"\")\n",
        "                elif event.event_type == \"citation-generation\":\n",
        "                    citations.extend(event.citations)\n",
        "                elif event.event_type == \"stream-end\":\n",
        "                    cited_documents = event.response.documents\n",
        "\n",
        "            # Display citations and source documents\n",
        "            if citations:\n",
        "              print(\"\\n\\nCITATIONS:\")\n",
        "              for citation in citations:\n",
        "                print(citation)\n",
        "\n",
        "              print(\"\\nDOCUMENTS:\")\n",
        "              for document in cited_documents:\n",
        "                print({'id': document['id'],\n",
        "                        'text': document['text'][:50] + '...'})\n",
        "\n",
        "            print(f\"\\n{'-'*100}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TAIvNbpxkA1h"
      },
      "source": [
        "# Run the chatbot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AL4D455vmED3"
      },
      "source": [
        "We can now run the chatbot.  For this, we create the instance of `Chatbot` using Cohere's managed web-search connector.  Then we run the chatbot by invoking the `run()` method.\n",
        "\n",
        "The format of each citation is:\n",
        "- `start`: The starting point of a span where one or more documents are referenced\n",
        "- `end`: The ending point of a span where one or more documents are referenced\n",
        "- `text`: The text representing this span\n",
        "- `document_ids`: The IDs of the documents being referenced (`doc_0` being the ID of the first document passed to the `documents` creating parameter in the endpoint call, and so on)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "JeEZkEVPRnmx",
        "outputId": "4e157035-f0cc-4f68-cde2-2404d92b230a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "User: What is prompt engineering\n",
            "\n",
            "Chatbot:\n",
            "Prompt engineering is a process of constructing prompts for a large language model (LLM) to get a desired response. It involves techniques such as adding specific details, context, output format requirements, and examples to a prompt to guide the LLM's response. Prompt engineering also involves chaining multiple prompts together to accomplish a complex task and validating and evaluating the outputs of LLMs to ensure the quality and accuracy of the responses.\n",
            "\n",
            "CITATIONS:\n",
            "start=35 end=55 text='constructing prompts' document_ids=['demo-conn-gdrive-6bfrp6_0', 'demo-conn-gdrive-6bfrp6_1', 'demo-conn-gdrive-6bfrp6_2', 'demo-conn-gdrive-6bfrp6_3', 'demo-conn-gdrive-6bfrp6_4']\n",
            "start=62 end=88 text='large language model (LLM)' document_ids=['demo-conn-gdrive-6bfrp6_0', 'demo-conn-gdrive-6bfrp6_3']\n",
            "start=98 end=115 text='desired response.' document_ids=['demo-conn-gdrive-6bfrp6_0', 'demo-conn-gdrive-6bfrp6_3']\n",
            "start=154 end=170 text='specific details' document_ids=['demo-conn-gdrive-6bfrp6_0']\n",
            "start=172 end=179 text='context' document_ids=['demo-conn-gdrive-6bfrp6_0']\n",
            "start=181 end=207 text='output format requirements' document_ids=['demo-conn-gdrive-6bfrp6_0']\n",
            "start=213 end=221 text='examples' document_ids=['demo-conn-gdrive-6bfrp6_0']\n",
            "start=237 end=262 text=\"guide the LLM's response.\" document_ids=['demo-conn-gdrive-6bfrp6_0']\n",
            "start=296 end=330 text='chaining multiple prompts together' document_ids=['demo-conn-gdrive-6bfrp6_1']\n",
            "start=364 end=374 text='validating' document_ids=['demo-conn-gdrive-6bfrp6_3']\n",
            "start=379 end=389 text='evaluating' document_ids=['demo-conn-gdrive-6bfrp6_4']\n",
            "start=424 end=431 text='quality' document_ids=['demo-conn-gdrive-6bfrp6_4']\n",
            "start=436 end=444 text='accuracy' document_ids=['demo-conn-gdrive-6bfrp6_3']\n",
            "\n",
            "DOCUMENTS:\n",
            "{'id': 'demo-conn-gdrive-6bfrp6_0', 'text': \"\\ufeffConstructing Prompts\\r\\nIn this chapter, you'll lea...\"}\n",
            "{'id': 'demo-conn-gdrive-6bfrp6_1', 'text': \"\\ufeffChaining Prompts\\r\\nIn this chapter, you'll learn a...\"}\n",
            "{'id': 'demo-conn-gdrive-6bfrp6_2', 'text': \"\\ufeffUse Case Patterns\\r\\nIn this chapter, you'll learn ...\"}\n",
            "{'id': 'demo-conn-gdrive-6bfrp6_3', 'text': \"\\ufeffValidating Outputs\\r\\nIn this chapter, you'll learn...\"}\n",
            "{'id': 'demo-conn-gdrive-6bfrp6_4', 'text': \"\\ufeffEvaluating Outputs\\r\\nIn this chapter, you'll learn...\"}\n",
            "\n",
            "----------------------------------------------------------------------------------------------------\n",
            "\n",
            "Ending chat.\n"
          ]
        }
      ],
      "source": [
        "# Define the connector\n",
        "connectors = [\"demo-conn-gdrive-6bfrp6\"]\n",
        "\n",
        "# Create an instance of the Chatbot class\n",
        "chatbot = Chatbot(connectors)\n",
        "\n",
        "# Run the chatbot\n",
        "chatbot.run()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}