From 22052f502d9ed77da13cd74d0938c556615ce6c5 Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Thu, 22 Feb 2024 12:50:34 -0600 Subject: [PATCH 1/8] feat: Add Grounding with Gemini Notebook --- gemini/grounding/intro-grounding.ipynb | 588 +++++++++++++++++++++++++ 1 file changed, 588 insertions(+) create mode 100644 gemini/grounding/intro-grounding.ipynb diff --git a/gemini/grounding/intro-grounding.ipynb b/gemini/grounding/intro-grounding.ipynb new file mode 100644 index 00000000000..9a15d7c1432 --- /dev/null +++ b/gemini/grounding/intro-grounding.ipynb @@ -0,0 +1,588 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# Copyright 2023 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JAPoU8Sm5E6e" + }, + "source": [ + "# Getting Started with Grounding with Gemini in Vertex AI\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Run in Colab\n", + " \n", + " 		\n", + " \n", + " \"Google
Run in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + " 		\n", + " \n", + " \"Vertex
Open in Vertex AI Workbench\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "| | |\n", + "|-|-|\n", + "|Author(s) | [Holt Skinner](https://github.com/holtskinner), [Kristopher Overholt](https://github.com/koverholt) |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "24743cf4a1e1" + }, + "source": [ + "**_NOTE_**: This notebook has been tested in the following environment:\n", + "\n", + "* Python version = 3.11" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "## Overview\n", + "\n", + "[Grounding in Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/grounding/ground-language-models) lets you use Gemini to generate content grounded in your own documents and data. This capability lets the model access information at runtime that goes beyond its training data. By grounding model responses in Google Search results or data stores within [Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction), LLMs that are grounded in data can produce more accurate, up-to-date, and relevant responses.\n", + "\n", + "Grounding provides the following benefits:\n", + "\n", + "- Reduces model hallucinations (instances where the model generates content that isn't factual)\n", + "- Anchors model responses to specific information, documents, and data sources\n", + "- Enhances the trustworthiness, accuracy, and applicability of the generated content\n", + "\n", + "In the context of grounding in Vertex AI, you can configure two different sources of grounding:\n", + "\n", + "1. Google Search results for data that is publicly available and indexed\n", + "2. [Data stores in Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/create-datastore-ingest), which can include your own data in the form of website data, unstructured data, or structured data\n", + "\n", + "**NOTE:** Some of the features in this sample notebook require early access to certain features via an allowlist. [Grounding with Vertex AI Search](https://cloud.google.com/vertex-ai/docs/generative-ai/grounding/ground-language-models) is available in Public Preview, whereas Grounding with Google Web Search results is available in Private Preview. To request early access to features in Private Preview, contact your account representative or [Google Cloud Support](https://cloud.google.com/contact)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "d975e698c9a4" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you learn how to:\n", + "\n", + "- Generate LLM text and chat model responses grounded in Google Search results\n", + "- Compare the results of ungrounded LLM responses with grounded LLM responses\n", + "- Create and use a data store in Vertex AI Search to ground responses in custom documents and data\n", + "- Generate LLM text and chat model responses grounded in Vertex AI Search results\n", + "- Use the asynchronous text and chat models APIs with grounding\n", + "\n", + "This tutorial uses the following Google Cloud AI services and resources:\n", + "\n", + "- Vertex AI\n", + "- Vertex AI Search and Conversation\n", + "\n", + "The steps performed include:\n", + "\n", + "- Configuring the LLM and prompt for various examples\n", + "- Sending example prompts to generative text and chat models in Vertex AI\n", + "- Setting up a data store in Vertex AI Search with your own data\n", + "- Sending example prompts with various levels of grounding (no grounding, web grounding, data store grounding)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BF1j6f9HApxa" + }, + "source": [ + "## Before you begin\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "1. Enable the [Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com) and [Vertex AI Search and Conversation API](https://console.cloud.google.com/flows/enableapi?apiid=discoveryengine.googleapis.com).\n", + "1. If you want to use Grounding with Google Web Search results, your project must also be allowlisted for this feature while it is in the Private Preview stage.\n", + "1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i7EUnXsZhAGF" + }, + "source": [ + "### Installation\n", + "\n", + "Install the following packages required to execute this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2b4ef9b72d43" + }, + "outputs": [], + "source": [ + "%pip install --upgrade --user --quiet google-cloud-aiplatform==1.42.1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "58707a750154" + }, + "source": [ + "Restart the kernel after installing packages:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "f200f10a1da3" + }, + "outputs": [], + "source": [ + "import IPython\n", + "\n", + "app = IPython.Application.instance()\n", + "app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WReHDGG5g0XY" + }, + "source": [ + "### Configure your project ID & region\n", + "\n", + "**If you don't know your project ID**, try the following:\n", + "* Run `gcloud config list`.\n", + "* Run `gcloud projects list`.\n", + "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)\n", + "\n", + "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oM1iC_MfAts1" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"your-project-id\" # @param {type:\"string\"}\n", + "REGION = \"us-central1\" # @param {type: \"string\"}\n", + "\n", + "# Set the project ID\n", + "!gcloud auth application-default set-quota-project {PROJECT_ID}\n", + "!gcloud config set project {PROJECT_ID}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sBCra4QMA2wR" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "If you are running this notebook on Google Colab, you will need to authenticate your environment. To do this, run the new cell below. This step is not required if you are using Vertex AI Workbench." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "603adbbf0532" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + " # Authenticate user to Google Cloud\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "960505627ddf" + }, + "source": [ + "### Import libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 63, + "metadata": { + "id": "PyQmSRbKA8r-" + }, + "outputs": [], + "source": [ + "import vertexai\n", + "from vertexai.preview.generative_models import GenerativeModel, Tool, grounding" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "init_aip:mbsdk,all" + }, + "source": [ + "### Initialize Vertex AI SDK for Python\n", + "\n", + "Initialize the Vertex AI SDK for Python for your project:" + ] + }, + { + "cell_type": "code", + "execution_count": 64, + "metadata": { + "id": "init_aip:mbsdk,all" + }, + "outputs": [], + "source": [ + "vertexai.init(project=PROJECT_ID, location=REGION)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Initialize the generative text and chat models from Vertex AI:" + ] + }, + { + "cell_type": "code", + "execution_count": 65, + "metadata": {}, + "outputs": [], + "source": [ + "model = GenerativeModel(\"gemini-1.0-pro\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example: Grounding with Google Search results\n", + "\n", + "In this example, you'll compare LLM responses with no grounding with responses that are grounded in the results of a Google Search. You'll ask a question about a recent hardware release from the Google Store." + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "metadata": {}, + "outputs": [], + "source": [ + "PROMPT = (\n", + " \"What are the price, available colors, and storage size options of a Pixel Tablet?\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Text generation without grounding\n", + "\n", + "Make a prediction request to the LLM with no grounding:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "response = model.generate_content(PROMPT)\n", + "response" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Text generation grounded in Google Search results\n", + "\n", + "Now you can add the `tools` keyword arg with a grounding tool of `grounding.GoogleSearchRetrieval()` to instruct the LLM to first perform a Google Search with the prompt, then construct an answer based on the web search results:" + ] + }, + { + "cell_type": "code", + "execution_count": 68, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "candidates {\n", + " content {\n", + " role: \"model\"\n", + " parts {\n", + " text: \"The Google Pixel Tablet is available in three colors: Porcelain, Rose, and Hazel. \\nIt has two storage size options: 128GB and 256GB. \\nThe price of the Pixel Tablet starts at $499 for the 128GB model and $599 for the 256GB model.\"\n", + " }\n", + " }\n", + " finish_reason: STOP\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HATE_SPEECH\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_DANGEROUS_CONTENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HARASSMENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_SEXUALLY_EXPLICIT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " grounding_metadata {\n", + " web_search_queries: \"Pixel Tablet price, colors, and storage size options?\"\n", + " grounding_attributions {\n", + " web {\n", + " uri: \"https://store.google.com/product/pixel_tablet_specs\"\n", + " title: \"Pixel Tablet Technical Specs - Google Store\"\n", + " }\n", + " segment {\n", + " end_index: 82\n", + " }\n", + " confidence_score: 0.882001281\n", + " }\n", + " grounding_attributions {\n", + " web {\n", + " uri: \"https://us.mobileinto.com/Google-Pixel-Tablet/\"\n", + " title: \"Google Pixel Tablet Price in USA February 2024 - Mobileinto\"\n", + " }\n", + " segment {\n", + " start_index: 83\n", + " end_index: 133\n", + " }\n", + " confidence_score: 0.922257721\n", + " }\n", + " }\n", + "}\n", + "usage_metadata {\n", + " prompt_token_count: 17\n", + " candidates_token_count: 74\n", + " total_token_count: 91\n", + "}" + ] + }, + "execution_count": 68, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tool = Tool.from_google_search_retrieval(grounding.GoogleSearchRetrieval())\n", + "\n", + "response = model.generate_content(PROMPT, tools=[tool])\n", + "\n", + "response" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that the response without grounding only has limited information from the LLM about the Pixel tablet. Whereas the response that was grounded in web search results contains the most up to date information from web search results that are returned as part of the LLM with grounding request." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example: Grounding with custom documents and data\n", + "\n", + "In this example, you'll compare LLM responses with no grounding with responses that are grounded in the [results of a data store in Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/create-datastore-ingest). You'll ask a question about a GoogleSQL query to create an [object table in BigQuery](https://cloud.google.com/bigquery/docs/object-table-introduction)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Creating a data store in Vertex AI Search\n", + "\n", + "Follow the steps in the [Vertex AI Search getting started documentation](https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#create_a_search_app_for_website_data) to create a data store in Vertex AI Search with sample data. In this example, you'll use a website-based data store that contains content from the Google Cloud website, including documentation.\n", + "\n", + "Note: The data store must be in the same project that you are using for Gemini.\n", + "\n", + "Once you've created a data store, obtain the Data Store ID and input it below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "DATA_STORE_ID = \"your-data-store-id_1234567890123\" # Replace this with your data store ID from Vertex AI Search\n", + "DATA_STORE_REGION = \"global\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now you can ask a question about object tables in BigQuery and when to use them:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "PROMPT = \"When should I use an object table in BigQuery? And how does it store data?\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Text generation without grounding\n", + "\n", + "Make a prediction request to the LLM with no grounding:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "response = model.generate_content(PROMPT)\n", + "\n", + "response" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Text generation grounded in Vertex AI Search results\n", + "\n", + "Now we can add the `grounding_source` keyword arg with a grounding source of `GroundingSource.VertexAISearch()` to instruct the LLM to first perform a search within your custom data store, then construct an answer based on the relevant documents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "datastore = f\"projects/{PROJECT_ID}/locations/{DATA_STORE_REGION}/collections/default_collection/dataStores/{DATA_STORE_ID}\"\n", + "tool = Tool.from_retrieval(\n", + " grounding.Retrieval(grounding.VertexAISearch(datastore=datastore))\n", + ")\n", + "\n", + "response = model.generate_content(PROMPT, tools=[tool])\n", + "\n", + "response.candidates[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that the response without grounding only has limited information from the LLM about object tables in BigQuery that might not be accurate. Whereas the response that was grounded in Vertex AI Search results contains the most up to date information from the Google Cloud documentation about BigQuery." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TpV-iwP9qw9c" + }, + "source": [ + "## Cleaning up\n", + "\n", + "To avoid incurring charges to your Google Cloud account for the resources used in this notebook, follow these steps:\n", + "\n", + "1. To avoid unnecessary Google Cloud charges, use the [Google Cloud console](https://console.cloud.google.com/) to delete your project if you do not need it. Learn more in the Google Cloud documentation for [managing and deleting your project](https://cloud.google.com/resource-manager/docs/creating-managing-projects).\n", + "1. If you used an existing Google Cloud project, delete the resources you created to avoid incurring charges to your account. For more information, refer to the documentation to [Delete data from a data store in Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/delete-datastores), then delete your data store.\n", + "1. Disable the [Vertex AI Search and Conversation API](https://pantheon.corp.google.com/apis/api/discoveryengine.googleapis.com) and [Vertex AI API](https://pantheon.corp.google.com/apis/api/aiplatform.googleapis.com) in the Google Cloud Console." + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "notebook_template.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} From a81f57ca392794aab20ab966881fa53d7250e16e Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Thu, 22 Feb 2024 12:50:45 -0600 Subject: [PATCH 2/8] Update gemini readme for Grounding and reorder --- gemini/README.md | 54 ++++++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 20 deletions(-) diff --git a/gemini/README.md b/gemini/README.md index 0b596a0817c..c555f827711 100644 --- a/gemini/README.md +++ b/gemini/README.md @@ -4,7 +4,9 @@ Welcome to the Google Cloud [Generative AI](https://cloud.google.com/ai/generati ## Gemini - + +Welcome to the Gemini era + Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases. The Gemini API gives you access to the Gemini Pro Vision and Gemini Pro models. @@ -12,7 +14,7 @@ Gemini is a family of generative AI models developed by Google DeepMind that is On Google Cloud, the Vertex AI Gemini API provides a unified interface for interacting with Gemini models. There are currently two models available in the Gemini API: -- **Gemini Pro model** (`gemini-pro`): Designed to handle natural language tasks, multiturn text and code chat, and code generation. +- **Gemini Pro model** (`gemini-pro`): Designed to handle natural language tasks, multi-turn text and code chat, and code generation. - **Gemini Pro Vision model** (`gemini-pro-vision`): Supports multimodal prompts. You can include text, images, and video in your prompt requests and get text or code responses. The notebooks and samples in this folder focus on using the **Vertex AI SDK for Python** to call the Vertex AI Gemini API. @@ -28,7 +30,7 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + flag
getting-started/ @@ -36,13 +38,13 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + + Starter notebooks - + deployed_code
sample-apps/ @@ -51,7 +53,7 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + manufacturing
use-cases/ @@ -62,7 +64,16 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + radar +
+ evaluation/ + + Learn how to evaluate Gemini with Vertex AI Model Evaluation for GenAI + Sample notebooks + + + + terminal
function-calling/ @@ -73,7 +84,18 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + grass +
+ grounding/ + + + Learn how to use the grounding feature of Gemini + + Sample notebooks + + + + health_and_safety
responsible-ai/ @@ -82,7 +104,7 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + build
setup-env/ @@ -91,21 +113,13 @@ The notebooks and samples in this folder focus on using the **Vertex AI SDK for - + media_link
RESOURCES.md Learning resources (e.g. blogs, Youtube playlists) about Generative AI on Google Cloud Resources (e.g. videos, blogposts, learning paths) - - - -
- RESOURCES.md - - Learn how to evaluate Gemini with Vertex AI Model Evaluation for GenAI - Sample notebooks - + From 94325a5333a5138f60e4c243afeff0bceb1d66ee Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Thu, 22 Feb 2024 12:53:14 -0600 Subject: [PATCH 3/8] Rename to intro-grounding-gemini --- .../{intro-grounding.ipynb => intro-grounding-gemini.ipynb} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename gemini/grounding/{intro-grounding.ipynb => intro-grounding-gemini.ipynb} (100%) diff --git a/gemini/grounding/intro-grounding.ipynb b/gemini/grounding/intro-grounding-gemini.ipynb similarity index 100% rename from gemini/grounding/intro-grounding.ipynb rename to gemini/grounding/intro-grounding-gemini.ipynb From 73ec625ea5947fef3d81b901cba785fa770591f8 Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Thu, 22 Feb 2024 13:19:36 -0600 Subject: [PATCH 4/8] Add markdown display and include cell outputs for grounding with Google Search --- gemini/grounding/intro-grounding-gemini.ipynb | 72 ++++++++++--------- 1 file changed, 40 insertions(+), 32 deletions(-) diff --git a/gemini/grounding/intro-grounding-gemini.ipynb b/gemini/grounding/intro-grounding-gemini.ipynb index 9a15d7c1432..277b288b64f 100644 --- a/gemini/grounding/intro-grounding-gemini.ipynb +++ b/gemini/grounding/intro-grounding-gemini.ipynb @@ -160,7 +160,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "id": "2b4ef9b72d43" }, @@ -210,7 +210,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "id": "oM1iC_MfAts1" }, @@ -263,14 +263,16 @@ }, { "cell_type": "code", - "execution_count": 63, + "execution_count": 15, "metadata": { "id": "PyQmSRbKA8r-" }, "outputs": [], "source": [ "import vertexai\n", - "from vertexai.preview.generative_models import GenerativeModel, Tool, grounding" + "from vertexai.preview.generative_models import GenerativeModel, Tool, grounding\n", + "\n", + "from IPython.display import display, Markdown" ] }, { @@ -286,7 +288,7 @@ }, { "cell_type": "code", - "execution_count": 64, + "execution_count": 3, "metadata": { "id": "init_aip:mbsdk,all" }, @@ -304,7 +306,7 @@ }, { "cell_type": "code", - "execution_count": 65, + "execution_count": 4, "metadata": {}, "outputs": [], "source": [ @@ -322,7 +324,7 @@ }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 18, "metadata": {}, "outputs": [], "source": [ @@ -342,12 +344,13 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "response = model.generate_content(PROMPT)\n", - "response" + "\n", + "display(Markdown(response.candidates[0].text))" ] }, { @@ -361,9 +364,23 @@ }, { "cell_type": "code", - "execution_count": 68, + "execution_count": 25, "metadata": {}, "outputs": [ + { + "data": { + "text/markdown": [ + "- Price: Starting from USD 499\n", + "- Storage size options: 128 GB and 256 GB\n", + "- Available colors: Porcelain, Rose, and Hazel" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, { "data": { "text/plain": [ @@ -371,7 +388,7 @@ " content {\n", " role: \"model\"\n", " parts {\n", - " text: \"The Google Pixel Tablet is available in three colors: Porcelain, Rose, and Hazel. \\nIt has two storage size options: 128GB and 256GB. \\nThe price of the Pixel Tablet starts at $499 for the 128GB model and $599 for the 256GB model.\"\n", + " text: \"- Price: Starting from USD 499\\n- Storage size options: 128 GB and 256 GB\\n- Available colors: Porcelain, Rose, and Hazel\"\n", " }\n", " }\n", " finish_reason: STOP\n", @@ -399,31 +416,20 @@ " title: \"Pixel Tablet Technical Specs - Google Store\"\n", " }\n", " segment {\n", - " end_index: 82\n", + " end_index: 119\n", " }\n", - " confidence_score: 0.882001281\n", - " }\n", - " grounding_attributions {\n", - " web {\n", - " uri: \"https://us.mobileinto.com/Google-Pixel-Tablet/\"\n", - " title: \"Google Pixel Tablet Price in USA February 2024 - Mobileinto\"\n", - " }\n", - " segment {\n", - " start_index: 83\n", - " end_index: 133\n", - " }\n", - " confidence_score: 0.922257721\n", + " confidence_score: 0.778106511\n", " }\n", " }\n", "}\n", "usage_metadata {\n", " prompt_token_count: 17\n", - " candidates_token_count: 74\n", - " total_token_count: 91\n", + " candidates_token_count: 38\n", + " total_token_count: 55\n", "}" ] }, - "execution_count": 68, + "execution_count": 25, "metadata": {}, "output_type": "execute_result" } @@ -433,6 +439,8 @@ "\n", "response = model.generate_content(PROMPT, tools=[tool])\n", "\n", + "display(Markdown(response.candidates[0].text))\n", + "\n", "response" ] }, @@ -467,7 +475,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 33, "metadata": {}, "outputs": [], "source": [ @@ -484,7 +492,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 30, "metadata": {}, "outputs": [], "source": [ @@ -502,13 +510,13 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "response = model.generate_content(PROMPT)\n", "\n", - "response" + "display(Markdown(response.candidates[0].text))" ] }, { @@ -522,7 +530,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 34, "metadata": {}, "outputs": [], "source": [ From ee42b7a03edaa01974eb363b201fce81be27d779 Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Tue, 27 Feb 2024 10:58:10 -0600 Subject: [PATCH 5/8] Update CODEOWNERS --- .github/CODEOWNERS | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index cfb4b226f07..a5f5c0316fa 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -11,13 +11,14 @@ /generative-ai/embeddings @kazunori279 @GoogleCloudPlatform/generative-ai-devrel /generative-ai/embeddings/use-cases/outlier-detection @rarsan @GoogleCloudPlatform/generative-ai-devrel /generative-ai/gemini @polong-lin @GoogleCloudPlatform/generative-ai-devrel +/generative-ai/gemini/evaluation @inardini @GoogleCloudPlatform/generative-ai-devrel /generative-ai/gemini/function-calling @koverholt @GoogleCloudPlatform/generative-ai-devrel +/generative-ai/gemini/getting-started @gericdong @polong-lin @GoogleCloudPlatform/generative-ai-devrel +/generative-ai/gemini/grounding @holtskinner @GoogleCloudPlatform/generative-ai-devrel +/generative-ai/gemini/responsible-ai @iamthuya @GoogleCloudPlatform/generative-ai-devrel /generative-ai/gemini/use-cases/education @PicardParis @GoogleCloudPlatform/generative-ai-devrel /generative-ai/gemini/use-cases/retail @iamthuya @GoogleCloudPlatform/generative-ai-devrel /generative-ai/gemini/use-cases/applying-llms-to-data/using-gemini-with-bigquery-remote-functions @shanecglass @GoogleCloudPlatform/generative-ai-devrel -/generative-ai/gemini/getting-started @gericdong @polong-lin @GoogleCloudPlatform/generative-ai-devrel -/generative-ai/gemini/evaluation @inardini @GoogleCloudPlatform/generative-ai-devrel -/generative-ai/gemini/responsible-ai @iamthuya @GoogleCloudPlatform/generative-ai-devrel /generative-ai/language @polong-lin @GoogleCloudPlatform/generative-ai-devrel /generative-ai/language/getting-started @polong-lin @GoogleCloudPlatform/generative-ai-devrel /generative-ai/language/grounding @koverholt @holtskinner @GoogleCloudPlatform/generative-ai-devrel From ecba7f8a6fb3f98128d0290a2d37a55e3f0cc0ea Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Tue, 27 Feb 2024 11:10:05 -0600 Subject: [PATCH 6/8] Minor changes - Added Restart warning - Added params arguments for data store - Added markdown display for vertex ai search results --- gemini/grounding/intro-grounding-gemini.ipynb | 59 ++++++++++++++----- 1 file changed, 43 insertions(+), 16 deletions(-) diff --git a/gemini/grounding/intro-grounding-gemini.ipynb b/gemini/grounding/intro-grounding-gemini.ipynb index 277b288b64f..eec668aafe2 100644 --- a/gemini/grounding/intro-grounding-gemini.ipynb +++ b/gemini/grounding/intro-grounding-gemini.ipynb @@ -160,13 +160,13 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": { "id": "2b4ef9b72d43" }, "outputs": [], "source": [ - "%pip install --upgrade --user --quiet google-cloud-aiplatform==1.42.1" + "%pip install --upgrade --user --quiet google-cloud-aiplatform>=1.42.1" ] }, { @@ -192,6 +192,15 @@ "app.kernel.do_shutdown(True)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": { @@ -263,7 +272,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 2, "metadata": { "id": "PyQmSRbKA8r-" }, @@ -324,7 +333,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 5, "metadata": {}, "outputs": [], "source": [ @@ -344,9 +353,24 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 6, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/markdown": [ + "As of my knowledge cutoff in April 2023, the Pixel Tablet has not yet been officially announced or released. Therefore, details such as pricing, color options, and storage size options are not yet available. \n", + "\n", + "However, you can check the official Google website or reputable tech news sources for the latest and most up-to-date information on the Pixel Tablet once it becomes available." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "response = model.generate_content(PROMPT)\n", "\n", @@ -364,7 +388,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 7, "metadata": {}, "outputs": [ { @@ -475,12 +499,13 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 8, "metadata": {}, "outputs": [], "source": [ - "DATA_STORE_ID = \"your-data-store-id_1234567890123\" # Replace this with your data store ID from Vertex AI Search\n", - "DATA_STORE_REGION = \"global\"" + "# Replace this with your data store ID from Vertex AI Search\n", + "DATA_STORE_ID = \"your-data-store-id_1234567890123\" # @param {type:\"string\"}\n", + "DATA_STORE_REGION = \"global\" # @param {type:\"string\"}" ] }, { @@ -492,7 +517,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -510,7 +535,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 10, "metadata": {}, "outputs": [], "source": [ @@ -525,12 +550,12 @@ "source": [ "### Text generation grounded in Vertex AI Search results\n", "\n", - "Now we can add the `grounding_source` keyword arg with a grounding source of `GroundingSource.VertexAISearch()` to instruct the LLM to first perform a search within your custom data store, then construct an answer based on the relevant documents:" + "Now we can add the `tools` keyword arg with a grounding tool of `grounding.VertexAISearch()` to instruct the LLM to first perform a search within your custom data store, then construct an answer based on the relevant documents:" ] }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 16, "metadata": {}, "outputs": [], "source": [ @@ -541,14 +566,16 @@ "\n", "response = model.generate_content(PROMPT, tools=[tool])\n", "\n", - "response.candidates[0]" + "display(Markdown(response.candidates[0].text))\n", + "\n", + "response" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Note that the response without grounding only has limited information from the LLM about object tables in BigQuery that might not be accurate. Whereas the response that was grounded in Vertex AI Search results contains the most up to date information from the Google Cloud documentation about BigQuery." + "Note that the response without grounding only has limited information from the LLM about object tables in BigQuery that might not be accurate. Whereas the response that was grounded in Vertex AI Search results contains the most up to date information from the Google Cloud documentation about BigQuery, along with citations of the information." ] }, { From 344a8058c1d05d9bee19b158aa2ada06d66a013f Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Tue, 27 Feb 2024 12:10:40 -0600 Subject: [PATCH 7/8] Addressed Review comments --- gemini/grounding/intro-grounding-gemini.ipynb | 578 +++++++++++++++++- 1 file changed, 563 insertions(+), 15 deletions(-) diff --git a/gemini/grounding/intro-grounding-gemini.ipynb b/gemini/grounding/intro-grounding-gemini.ipynb index eec668aafe2..a6de761374a 100644 --- a/gemini/grounding/intro-grounding-gemini.ipynb +++ b/gemini/grounding/intro-grounding-gemini.ipynb @@ -8,7 +8,7 @@ }, "outputs": [], "source": [ - "# Copyright 2023 Google LLC\n", + "# Copyright 2024 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", @@ -83,7 +83,7 @@ "source": [ "## Overview\n", "\n", - "[Grounding in Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/grounding/ground-language-models) lets you use Gemini to generate content grounded in your own documents and data. This capability lets the model access information at runtime that goes beyond its training data. By grounding model responses in Google Search results or data stores within [Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction), LLMs that are grounded in data can produce more accurate, up-to-date, and relevant responses.\n", + "[Grounding in Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/grounding/ground-language-models) lets you use generative text models to generate content grounded in your own documents and data. This capability lets the model access information at runtime that goes beyond its training data. By grounding model responses in Google Search results or data stores within [Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction), LLMs that are grounded in data can produce more accurate, up-to-date, and relevant responses.\n", "\n", "Grounding provides the following benefits:\n", "\n", @@ -166,7 +166,7 @@ }, "outputs": [], "source": [ - "%pip install --upgrade --user --quiet google-cloud-aiplatform>=1.42.1" + "%pip install --upgrade --user --quiet google-cloud-aiplatform==1.42.1" ] }, { @@ -219,7 +219,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 15, "metadata": { "id": "oM1iC_MfAts1" }, @@ -297,7 +297,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 16, "metadata": { "id": "init_aip:mbsdk,all" }, @@ -315,7 +315,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 17, "metadata": {}, "outputs": [], "source": [ @@ -499,13 +499,14 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": {}, "outputs": [], "source": [ + "DATA_STORE_PROJECT_ID = PROJECT_ID # @param {type:\"string\"}\n", + "DATA_STORE_REGION = \"global\" # @param {type:\"string\"}\n", "# Replace this with your data store ID from Vertex AI Search\n", - "DATA_STORE_ID = \"your-data-store-id_1234567890123\" # @param {type:\"string\"}\n", - "DATA_STORE_REGION = \"global\" # @param {type:\"string\"}" + "DATA_STORE_ID = \"your-data-store-id_1234567890123\" # @param {type:\"string\"}" ] }, { @@ -517,7 +518,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 6, "metadata": {}, "outputs": [], "source": [ @@ -535,9 +536,76 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 8, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/markdown": [ + "**When to Use an Object Table:**\n", + "\n", + "Use an object table in BigQuery when you need to store and query structured data in a semi-nested format, particularly when:\n", + "\n", + "* You have data with complex or hierarchical relationships.\n", + "* You want to efficiently query data across multiple levels of nesting.\n", + "* You need to represent parent-child relationships in your data.\n", + "* You want to model data that naturally fits into a hierarchical or tree-like structure.\n", + "\n", + "**How Object Tables Store Data:**\n", + "\n", + "Object tables store data as a collection of objects. Each object can have a unique set of fields and can be nested within other objects. The data structure is represented using a JSON-like format called **Arrow Flight Schema**.\n", + "\n", + "An object table can have the following elements:\n", + "\n", + "* **Root Object:** The top-level object in the table.\n", + "* **Nested Objects:** Objects within other objects, representing hierarchical relationships.\n", + "* **Primitive Fields:** Fields that store basic data types (e.g., string, number, date).\n", + "* **Complex Fields:** Fields that contain nested objects or arrays of objects.\n", + "* **Arrays:** Ordered lists of objects or primitive values.\n", + "\n", + "**Example:**\n", + "\n", + "Consider the following data representing employees and their departments:\n", + "\n", + "```json\n", + "{\n", + " \"employees\": [\n", + " {\n", + " \"name\": \"Alice\",\n", + " \"department\": {\n", + " \"name\": \"Engineering\",\n", + " \"manager\": \"Bob\"\n", + " }\n", + " },\n", + " {\n", + " \"name\": \"Bob\",\n", + " \"department\": {\n", + " \"name\": \"Management\",\n", + " \"manager\": \"Carol\"\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "```\n", + "\n", + "This data can be represented in an object table as follows:\n", + "\n", + "* **Root Object:** `employees` (an array of employee objects)\n", + "* **Nested Object:** `department` (an object representing the department for each employee)\n", + "* **Primitive Fields:** `name` (employee name), `manager` (department manager)\n", + "* **Complex Field:** `department` (contains nested objects)\n", + "* **Array:** `employees` (a collection of employee objects)\n", + "\n", + "By using an object table, you can easily query data across different levels of nesting, such as finding all employees in the \"Engineering\" department or the manager of \"Bob\"." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "response = model.generate_content(PROMPT)\n", "\n", @@ -555,11 +623,86 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 9, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/markdown": [ + "Here are the scenarios when you should use an object table in BigQuery:\n", + "\n", + "* **When you need to analyze unstructured data in Cloud Storage.** \n", + "Object tables let you analyze unstructured data in Cloud Storage, such as images, videos, and audio files. \n", + "* **When you need to perform analysis with remote functions.** \n", + "Object tables let you perform analysis with remote functions, such as image classification and speech recognition. \n", + "* **When you need to perform inference by using BigQuery ML.** \n", + "Object tables let you perform inference by using BigQuery ML, such as predicting the sentiment of text or the genre of music.\n", + "\n", + "BigQuery object tables store data in Cloud Storage. The data is stored in a columnar format, which makes it efficient for querying. Object tables also support partitioning, which can improve query performance." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "candidates {\n", + " content {\n", + " role: \"model\"\n", + " parts {\n", + " text: \"Here are the scenarios when you should use an object table in BigQuery:\\n\\n* **When you need to analyze unstructured data in Cloud Storage.** \\nObject tables let you analyze unstructured data in Cloud Storage, such as images, videos, and audio files. \\n* **When you need to perform analysis with remote functions.** \\nObject tables let you perform analysis with remote functions, such as image classification and speech recognition. \\n* **When you need to perform inference by using BigQuery ML.** \\nObject tables let you perform inference by using BigQuery ML, such as predicting the sentiment of text or the genre of music.\\n\\nBigQuery object tables store data in Cloud Storage. The data is stored in a columnar format, which makes it efficient for querying. Object tables also support partitioning, which can improve query performance.\"\n", + " }\n", + " }\n", + " finish_reason: STOP\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HATE_SPEECH\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_DANGEROUS_CONTENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HARASSMENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_SEXUALLY_EXPLICIT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " grounding_metadata {\n", + " web_search_queries: \"When should I use an object table in BigQuery?\"\n", + " grounding_attributions {\n", + " web {\n", + " uri: \"https://cloud.google.com/bigquery/docs/object-table-introduction\"\n", + " title: \"Introduction to object tables | BigQuery | Google Cloud\"\n", + " }\n", + " segment {\n", + " start_index: 429\n", + " end_index: 488\n", + " }\n", + " confidence_score: 0.607541919\n", + " }\n", + " }\n", + "}\n", + "usage_metadata {\n", + " prompt_token_count: 18\n", + " candidates_token_count: 169\n", + " total_token_count: 187\n", + "}" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "datastore = f\"projects/{PROJECT_ID}/locations/{DATA_STORE_REGION}/collections/default_collection/dataStores/{DATA_STORE_ID}\"\n", + "datastore = f\"projects/{DATA_STORE_PROJECT_ID}/locations/{DATA_STORE_REGION}/collections/default_collection/dataStores/{DATA_STORE_ID}\"\n", "tool = Tool.from_retrieval(\n", " grounding.Retrieval(grounding.VertexAISearch(datastore=datastore))\n", ")\n", @@ -578,6 +721,411 @@ "Note that the response without grounding only has limited information from the LLM about object tables in BigQuery that might not be accurate. Whereas the response that was grounded in Vertex AI Search results contains the most up to date information from the Google Cloud documentation about BigQuery, along with citations of the information." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example: Grounded chat responses\n", + "\n", + "You can also use grounding when working with chat models in Vertex AI. In this example, you'll compare LLM responses with no grounding with responses that are grounded in the results of a Google Search and a data store in Vertex AI Search.\n", + "\n", + "You'll ask a question about Vertex AI and a follow up question about managed datasets in Vertex AI:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "PROMPT = \"What are managed datasets in Vertex AI?\"\n", + "PROMPT_FOLLOWUP = \"What types of data can I use?\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Chat session without grounding\n", + "\n", + "Start a chat session and send messages to the LLM with no grounding:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "Managed datasets in Vertex AI are scalable, high-performance datasets that can be easily accessed and managed by AI and machine learning models. These datasets are hosted on Google Cloud Storage and are automatically managed by Vertex AI. \n", + "\n", + "Managed datasets are designed to simplify the process of working with large and complex datasets. This allows data scientists and machine learning engineers to focus on building and training models, rather than managing the underlying infrastructure.\n", + "\n", + "Managed datasets also provide a secure and scalable way to share datasets with collaborators. With managed datasets, users can grant access to specific users or groups, and can easily revoke access when needed. \n", + "\n", + "Overall, managed datasets are a valuable tool for data scientists and machine learning engineers who are looking to simplify the process of working with large and complex datasets." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/markdown": [ + "You can use a wide variety of data types with managed datasets in Vertex AI, including:\n", + "\n", + "* **Structured data:** This type of data is organized into rows and columns, and can be easily imported from CSV, JSON, or BigQuery files. Examples of structured data include customer data, sales data, and financial data.\n", + "* **Unstructured data:** This type of data is not organized into a specific format, and can include text, images, audio, and video. Examples of unstructured data include social media posts, product reviews, and medical records.\n", + "* **Semi-structured data:** This type of data has some structure, but is not as rigidly organized as structured data. Examples of semi-structured data include JSON and XML files.\n", + "\n", + "Managed datasets also support a variety of data formats, including:\n", + "\n", + "* CSV\n", + "* JSON\n", + "* BigQuery\n", + "* Apache Parquet\n", + "* Apache Avro\n", + "\n", + "You can also use managed datasets to import data from Google Cloud Storage, BigQuery, and other data sources.\n", + "\n", + "Once you have imported your data into a managed dataset, you can use it to train and evaluate machine learning models. Managed datasets are designed to be scalable and high-performance, so you can use them to train even the most complex models.\n", + "\n", + "Here are some examples of how you can use managed datasets in Vertex AI:\n", + "\n", + "* Train a machine learning model to predict customer churn using structured data from a CRM system.\n", + "* Train a machine learning model to classify images of products using unstructured data from a product catalog.\n", + "* Train a machine learning model to translate text from one language to another using semi-structured data from a translation corpus.\n", + "\n", + "Managed datasets are a versatile and powerful tool that can be used to train and evaluate machine learning models on a wide variety of data types and formats." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "chat = model.start_chat()\n", + "\n", + "response = chat.send_message(PROMPT)\n", + "display(Markdown(response.text))\n", + "\n", + "response = chat.send_message(PROMPT_FOLLOWUP)\n", + "display(Markdown(response.text))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Chat session grounded in Google Search results\n", + "\n", + "Now you can add the `tools` keyword arg with a grounding tool of `grounding.GoogleSearchRetrieval()` to instruct the chat model to first perform a Google Search with the prompt, then construct an answer based on the web search results:" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "Managed datasets in Vertex AI are a fully managed data service that provides a central repository for your training and prediction data. It offers features like data versioning, schema management, and data quality monitoring. Managed datasets also simplify data sharing and collaboration within your organization." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "candidates {\n", + " content {\n", + " role: \"model\"\n", + " parts {\n", + " text: \"Managed datasets in Vertex AI are a fully managed data service that provides a central repository for your training and prediction data. It offers features like data versioning, schema management, and data quality monitoring. Managed datasets also simplify data sharing and collaboration within your organization.\"\n", + " }\n", + " }\n", + " finish_reason: STOP\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HATE_SPEECH\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_DANGEROUS_CONTENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HARASSMENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_SEXUALLY_EXPLICIT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " grounding_metadata {\n", + " web_search_queries: \"What are managed datasets in Vertex AI?\"\n", + " }\n", + "}\n", + "usage_metadata {\n", + " prompt_token_count: 8\n", + " candidates_token_count: 51\n", + " total_token_count: 59\n", + "}\n", + "\n" + ] + }, + { + "data": { + "text/markdown": [ + "Managed datasets in Vertex AI support the following data types:\n", + "\n", + "* **Tabular data:** Data in a tabular format, such as CSV or BigQuery tables.\n", + "* **Image data:** Images in JPEG, PNG, or GIF format.\n", + "* **Video data:** Videos in MP4 or MOV format.\n", + "* **Text data:** Text files in TXT or JSON format." + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "candidates {\n", + " content {\n", + " role: \"model\"\n", + " parts {\n", + " text: \"Managed datasets in Vertex AI support the following data types:\\n\\n* **Tabular data:** Data in a tabular format, such as CSV or BigQuery tables.\\n* **Image data:** Images in JPEG, PNG, or GIF format.\\n* **Video data:** Videos in MP4 or MOV format.\\n* **Text data:** Text files in TXT or JSON format.\"\n", + " }\n", + " }\n", + " finish_reason: STOP\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HATE_SPEECH\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_DANGEROUS_CONTENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HARASSMENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_SEXUALLY_EXPLICIT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " grounding_metadata {\n", + " web_search_queries: \"Vertex AI managed datasets\"\n", + " }\n", + "}\n", + "usage_metadata {\n", + " prompt_token_count: 67\n", + " candidates_token_count: 76\n", + " total_token_count: 143\n", + "}\n", + "\n" + ] + } + ], + "source": [ + "chat = model.start_chat()\n", + "tool = Tool.from_google_search_retrieval(grounding.GoogleSearchRetrieval())\n", + "\n", + "response = chat.send_message(PROMPT, tools=[tool])\n", + "display(Markdown(response.text))\n", + "print(response)\n", + "\n", + "response = chat.send_message(PROMPT_FOLLOWUP, tools=[tool])\n", + "display(Markdown(response.text))\n", + "print(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Chat session grounded in Vertex AI Search results\n", + "\n", + "Now we can add the `tools` keyword arg with a grounding tool of `grounding.VertexAISearch()` to instruct the chat model to first perform a search within your custom data store, then construct an answer based on the relevant documents:" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "Managed datasets in Vertex AI are datasets that are stored and managed by Google Cloud. This means that you don't have to worry about the underlying infrastructure or the management of the data. You can simply access the data through the Vertex AI API or UI.\n", + "\n", + "Managed datasets are a good option for those who want to focus on building and training models without having to worry about the data management. They are also a good option for those who need to share data with others in a secure and controlled way.\n", + "\n", + "Here are some of the benefits of using managed datasets in Vertex AI:\n", + "\n", + "* **Reduced operational overhead:** You don't have to worry about the underlying infrastructure or the management of the data.\n", + "* **Improved security:** Managed datasets are stored in a secure and controlled environment.\n", + "* **Increased collaboration:** You can easily share data with others in a secure and controlled way.\n", + "* **Simplified data management:** You can easily manage your data through the Vertex AI API or UI.\n", + "\n", + "If you are interested in using managed datasets in Vertex AI, you can learn more here: https://cloud.google.com/vertex-ai/docs/datasets/overview" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "candidates {\n", + " content {\n", + " role: \"model\"\n", + " parts {\n", + " text: \"Managed datasets in Vertex AI are datasets that are stored and managed by Google Cloud. This means that you don\\'t have to worry about the underlying infrastructure or the management of the data. You can simply access the data through the Vertex AI API or UI.\\n\\nManaged datasets are a good option for those who want to focus on building and training models without having to worry about the data management. They are also a good option for those who need to share data with others in a secure and controlled way.\\n\\nHere are some of the benefits of using managed datasets in Vertex AI:\\n\\n* **Reduced operational overhead:** You don\\'t have to worry about the underlying infrastructure or the management of the data.\\n* **Improved security:** Managed datasets are stored in a secure and controlled environment.\\n* **Increased collaboration:** You can easily share data with others in a secure and controlled way.\\n* **Simplified data management:** You can easily manage your data through the Vertex AI API or UI.\\n\\nIf you are interested in using managed datasets in Vertex AI, you can learn more here: https://cloud.google.com/vertex-ai/docs/datasets/overview\"\n", + " }\n", + " }\n", + " finish_reason: STOP\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HATE_SPEECH\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_DANGEROUS_CONTENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HARASSMENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_SEXUALLY_EXPLICIT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " grounding_metadata {\n", + " web_search_queries: \"What are managed datasets in Vertex AI?\"\n", + " }\n", + "}\n", + "usage_metadata {\n", + " prompt_token_count: 8\n", + " candidates_token_count: 234\n", + " total_token_count: 242\n", + "}\n", + "\n" + ] + }, + { + "data": { + "text/markdown": [ + "With managed datasets in Vertex AI, you can use the following types of data:\n", + "\n", + "* Tabular data\n", + "* Image data\n", + "* Video data\n", + "* Text data\n", + "* Time series data\n", + "\n", + "You can also use managed datasets to store and manage data that is used for training and deploying models. This data can include:\n", + "\n", + "* Training data\n", + "* Test data\n", + "* Validation data\n", + "* Model artifacts\n", + "* Predictions\n", + "\n", + "Managed datasets are a good option for storing and managing data that is used for machine learning because they are:\n", + "\n", + "* **Secure:** Data is stored in a secure and controlled environment.\n", + "* **Scalable:** Datasets can be scaled to accommodate large amounts of data.\n", + "* **Flexible:** Datasets can be used for a variety of machine learning tasks.\n", + "\n", + "To learn more about managed datasets in Vertex AI, you can visit the following link: https://cloud.google.com/vertex-ai/docs/datasets/overview" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "candidates {\n", + " content {\n", + " role: \"model\"\n", + " parts {\n", + " text: \"With managed datasets in Vertex AI, you can use the following types of data:\\n\\n* Tabular data\\n* Image data\\n* Video data\\n* Text data\\n* Time series data\\n\\nYou can also use managed datasets to store and manage data that is used for training and deploying models. This data can include:\\n\\n* Training data\\n* Test data\\n* Validation data\\n* Model artifacts\\n* Predictions\\n\\nManaged datasets are a good option for storing and managing data that is used for machine learning because they are:\\n\\n* **Secure:** Data is stored in a secure and controlled environment.\\n* **Scalable:** Datasets can be scaled to accommodate large amounts of data.\\n* **Flexible:** Datasets can be used for a variety of machine learning tasks.\\n\\nTo learn more about managed datasets in Vertex AI, you can visit the following link: https://cloud.google.com/vertex-ai/docs/datasets/overview\"\n", + " }\n", + " }\n", + " finish_reason: STOP\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HATE_SPEECH\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_DANGEROUS_CONTENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_HARASSMENT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " safety_ratings {\n", + " category: HARM_CATEGORY_SEXUALLY_EXPLICIT\n", + " probability: NEGLIGIBLE\n", + " }\n", + " grounding_metadata {\n", + " web_search_queries: \"What types of data can I use with managed datasets in Vertex AI?\"\n", + " }\n", + "}\n", + "usage_metadata {\n", + " prompt_token_count: 250\n", + " candidates_token_count: 189\n", + " total_token_count: 439\n", + "}\n", + "\n" + ] + } + ], + "source": [ + "chat = model.start_chat()\n", + "datastore = f\"projects/{DATA_STORE_PROJECT_ID}/locations/{DATA_STORE_REGION}/collections/default_collection/dataStores/{DATA_STORE_ID}\"\n", + "tool = Tool.from_retrieval(\n", + " grounding.Retrieval(grounding.VertexAISearch(datastore=datastore))\n", + ")\n", + "\n", + "response = chat.send_message(PROMPT, tools=[tool])\n", + "display(Markdown(response.text))\n", + "print(response)\n", + "\n", + "response = chat.send_message(PROMPT_FOLLOWUP, tools=[tool])\n", + "display(Markdown(response.text))\n", + "print(response)" + ] + }, { "cell_type": "markdown", "metadata": { From 299f68680619f2d545371404e2b70036163a62f9 Mon Sep 17 00:00:00 2001 From: Holt Skinner Date: Tue, 27 Feb 2024 12:11:16 -0600 Subject: [PATCH 8/8] Removed bullet about async --- gemini/grounding/intro-grounding-gemini.ipynb | 1 - 1 file changed, 1 deletion(-) diff --git a/gemini/grounding/intro-grounding-gemini.ipynb b/gemini/grounding/intro-grounding-gemini.ipynb index a6de761374a..d89eda73db8 100644 --- a/gemini/grounding/intro-grounding-gemini.ipynb +++ b/gemini/grounding/intro-grounding-gemini.ipynb @@ -113,7 +113,6 @@ "- Compare the results of ungrounded LLM responses with grounded LLM responses\n", "- Create and use a data store in Vertex AI Search to ground responses in custom documents and data\n", "- Generate LLM text and chat model responses grounded in Vertex AI Search results\n", - "- Use the asynchronous text and chat models APIs with grounding\n", "\n", "This tutorial uses the following Google Cloud AI services and resources:\n", "\n",