Add OpenAI Gemini Example for VertexAI Notebook

website/docs/topics/non-openai-models/cloud-gemini_vertexai.ipynb

@@ -62,7 +62,12 @@
     "<img src=\"https://github.com/microsoft/autogen/blob/main/website/static/img/create_gcp_svc.png?raw=true\" width=\"1000\" />\n",
     "</div>\n",
     "\n",
-    "For the sake of simplicity we will assign the Editor role to our service account for autogen on our Autogen-with-Gemini Google Cloud project.\n",
+    "Next we assign the [Vertex AI User](https://cloud.google.com/vertex-ai/docs/general/access-control#aiplatform.user) for the service account. This can be done in the [Google Cloud console](https://console.cloud.google.com/iam-admin/iam?project=autogen-with-gemini) in our `autogen-with-gemini` project.<br/>\n",
+    "Alternatively, we can also grant the [Vertex AI User](https://cloud.google.com/vertex-ai/docs/general/access-control#aiplatform.user) role by running a command using the gcloud CLI, for example in [Cloud Shell](https://shell.cloud.google.com/cloudshell):\n",
+    "```bash\n",
+    "gcloud projects add-iam-policy-binding autogen-with-gemini \\\n",
+    "    --member=serviceAccount:[email protected] --role roles/aiplatform.user\n",
+    "```\n",
     "\n",
     "* Under IAM & Admin > Service Account select the newly created service accounts, and click the option \"Manage keys\" among the items. \n",
     "* From the \"ADD KEY\" dropdown select \"Create new key\" and select the JSON format and click CREATE.\n",
@@ -83,23 +88,23 @@
     "Additionally, AutoGen also supports authentication using `Credentials` objects in Python with the [google-auth library](https://google-auth.readthedocs.io/), which enables even more flexibility.<br/>\n",
     "For example, we can even use impersonated credentials.\n",
     "\n",
-    "#### Use Service Account Keyfile\n",
+    "#### <a id='use_svc_keyfile'></a>Use Service Account Keyfile\n",
     "\n",
     "The Google Cloud service account can be specified by setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path to the JSON key file of the service account. <br/>\n",
     "\n",
     "We could even just directly set the environment variable, or we can add the `\"google_application_credentials\"` key with the respective value for our model in the OAI_CONFIG_LIST.\n",
     "\n",
     "#### Use the Google Default Credentials\n",
     "\n",
-    "If you are using [Cloud Shell](https://shell.cloud.google.com/cloudshell) or [Cloud Shell editor](https://shell.cloud.google.com/cloudshell/editor) in Google Cloud, <br/> then you are already authenticated. If you have the Google Cloud SDK installed locally,  <br/> then you can login by running `gcloud auth login` in the command line. \n",
+    "If you are using [Cloud Shell](https://shell.cloud.google.com/cloudshell) or [Cloud Shell editor](https://shell.cloud.google.com/cloudshell/editor) in Google Cloud, <br/> then you are already authenticated. If you have the Google Cloud SDK installed locally,  <br/> then you can login by running `gcloud auth application-default login` in the command line. \n",
     "\n",
     "Detailed instructions for installing the Google Cloud SDK can be found [here](https://cloud.google.com/sdk/docs/install).\n",
     "\n",
     "#### Authentication with the Google Auth Library for Python\n",
     "\n",
     "The google-auth library supports a wide range of authentication scenarios, and you can simply pass a previously created `Credentials` object to the `llm_config`.<br/>\n",
     "The [official documentation](https://google-auth.readthedocs.io/) of the Python package provides a detailed overview of the supported methods and usage examples.<br/>\n",
-    "If you are already authenticated, like in [Cloud Shell](https://shell.cloud.google.com/cloudshell), or after running the `gcloud auth login` command in a CLI, then the `google.auth.default()` Python method will automatically return your currently active credentials."
+    "If you are already authenticated, like in [Cloud Shell](https://shell.cloud.google.com/cloudshell), or after running the `gcloud auth application-default login` command in a CLI, then the `google.auth.default()` Python method will automatically return your currently active credentials."
    ]
   },
   {
@@ -307,13 +312,13 @@
     "gcloud auth application-default login\n",
     "gcloud config set project autogen-with-gemini\n",
     "```\n",
-    "The `GOOGLE_APPLICATION_CREDENTIALS` environment variable is a path to our service account JSON keyfile, as described in the [Use Service Account Keyfile](#Use Service Account Keyfile) section above.<br/>\n",
+    "The `GOOGLE_APPLICATION_CREDENTIALS` environment variable is a path to our service account JSON keyfile, as described in the [Use Service Account Keyfile](#use_svc_keyfile) section above.<br/>\n",
     "We also need to set the Google cloud project, which is `autogen-with-gemini` in this example.<br/><br/>\n",
     "\n",
-    "Note, we could also run `gcloud auth login` in case we wish to use our personal Google account instead of a service account.\n",
+    "Note, we could also run `gcloud auth application-default login` to use our personal Google account instead of a service account.\n",
     "In this case we need to run the following commands:\n",
     "```bash\n",
-    "gcloud auth login\n",
+    "gcloud gcloud auth application-default login\n",
     "gcloud config set project autogen-with-gemini\n",
     "```"
    ]
@@ -390,6 +395,132 @@
     "<img https://github.com/microsoft/autogen/blob/main/website/static/img/autogen_agentchat.png?raw=true>.\"\"\",\n",
     ")"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Use Gemini via the OpenAI Library in Autogen\n",
+    "Using Gemini via the OpenAI library is also possible once you are already authenticated. <br/>\n",
+    "Run `gcloud auth application-default login` to set up application default credentials locally for the example below.<br/>\n",
+    "Also set the Google cloud project on the CLI if you have not done so far: <br/>\n",
+    "```bash\n",
+    "gcloud config set project autogen-with-gemini\n",
+    "```\n",
+    "The prerequisites are essentially the same as in the example above.<br/>\n",
+    "\n",
+    "You can read more on the topic in the [official Google docs](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library).\n",
+    "<br/> A list of currently supported models can also be found in the [docs](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library#supported_models)\n",
+    "<br/>\n",
+    "<br/>\n",
+    "Note, that you will need to refresh your token regularly, by default every 1 hour."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import google.auth\n",
+    "\n",
+    "scopes = [\"https://www.googleapis.com/auth/cloud-platform\"]\n",
+    "creds, project = google.auth.default(scopes)\n",
+    "auth_req = google.auth.transport.requests.Request()\n",
+    "creds.refresh(auth_req)\n",
+    "location = \"us-west1\"\n",
+    "prompt_price_per_1k = (\n",
+    "    0.000125  # For more up-to-date prices see https://cloud.google.com/vertex-ai/generative-ai/pricing\n",
+    ")\n",
+    "completion_token_price_per_1k = (\n",
+    "    0.000375  # For more up-to-date prices see https://cloud.google.com/vertex-ai/generative-ai/pricing\n",
+    ")\n",
+    "\n",
+    "openai_gemini_config = [\n",
+    "    {\n",
+    "        \"model\": \"google/gemini-1.5-pro-001\",\n",
+    "        \"api_type\": \"openai\",\n",
+    "        \"base_url\": f\"https://{location}-aiplatform.googleapis.com/v1beta1/projects/{project}/locations/{location}/endpoints/openapi\",\n",
+    "        \"api_key\": creds.token,\n",
+    "        \"price\": [prompt_price_per_1k, completion_token_price_per_1k],\n",
+    "    }\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
+      "\n",
+      "\n",
+      "    Compute the integral of the function f(x)=x^3 on the interval 0 to 10 using a Python script,\n",
+      "    which returns the value of the definite integral.\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33massistant\u001b[0m (to user_proxy):\n",
+      "\n",
+      "```python\n",
+      "# filename: integral.py\n",
+      "def integrate_x_cubed(a, b):\n",
+      "  \"\"\"\n",
+      "  This function calculates the definite integral of x^3 from a to b.\n",
+      "\n",
+      "  Args:\n",
+      "      a: The lower limit of integration.\n",
+      "      b: The upper limit of integration.\n",
+      "\n",
+      "  Returns:\n",
+      "      The value of the definite integral.\n",
+      "  \"\"\"\n",
+      "  return (b**4 - a**4) / 4\n",
+      "\n",
+      "# Calculate the integral of x^3 from 0 to 10\n",
+      "result = integrate_x_cubed(0, 10)\n",
+      "\n",
+      "# Print the result\n",
+      "print(result)\n",
+      "```\n",
+      "\n",
+      "This script defines a function `integrate_x_cubed` that takes the lower and upper limits of integration as arguments and returns the definite integral of x^3 using the power rule of integration. The script then calls this function with the limits 0 and 10 and prints the result.\n",
+      "\n",
+      "Execute the script `python integral.py`, you should get the result: `2500.0`.\n",
+      "\n",
+      "TERMINATE\n",
+      "\n",
+      "\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "assistant = AssistantAgent(\"assistant\", llm_config={\"config_list\": openai_gemini_config}, max_consecutive_auto_reply=3)\n",
+    "\n",
+    "user_proxy = UserProxyAgent(\n",
+    "    \"user_proxy\",\n",
+    "    code_execution_config={\"work_dir\": \"coding\", \"use_docker\": False},\n",
+    "    human_input_mode=\"NEVER\",\n",
+    "    is_termination_msg=lambda x: content_str(x.get(\"content\")).find(\"TERMINATE\") >= 0,\n",
+    ")\n",
+    "\n",
+    "result = user_proxy.initiate_chat(\n",
+    "    assistant,\n",
+    "    message=\"\"\"\n",
+    "    Compute the integral of the function f(x)=x^3 on the interval 0 to 10 using a Python script,\n",
+    "    which returns the value of the definite integral.\"\"\",\n",
+    ")"
+   ]
   }
  ],
  "metadata": {