cookbooks/05-video-processing-with-natural-language.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\" id=\"top\">\n",
    "<img src=\"https://socialify.git.ci/julep-ai/julep/image?description=1&descriptionEditable=Serverless%20AI%20Workflows%20for%20Data%20%26%20ML%20Teams&font=Source%20Code%20Pro&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fjulep-ai%2Fjulep%2Fdev%2F.github%2Fjulep-logo.svg&owner=1&forks=1&pattern=Solid&stargazers=1&theme=Auto\" alt=\"julep\" />\n",
    "\n",
    "<br>\n",
    "  <p>\n",
    "    <a href=\"https://dashboard.julep.ai\">\n",
    "      <img src=\"https://img.shields.io/badge/Get_API_Key-FF5733?style=logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0id2hpdGUiPjxwYXRoIGQ9Ik0xMiAxTDMgNXYxNGw5IDQgOS00VjVsLTktNHptMCAyLjh2MTYuNEw1IDE2LjJWNi44bDctMy4yem0yIDguMmwtMi0yLTIgMiAyIDIgMi0yeiIvPjwvc3ZnPg==\" alt=\"Get API Key\" height=\"28\">\n",
    "    </a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://docs.julep.ai\">\n",
    "      <img src=\"https://img.shields.io/badge/Documentation-4B32C3?style=logo=gitbook&logoColor=white\" alt=\"Documentation\" height=\"28\">\n",
    "    </a>\n",
    "  </p>\n",
    "  <p>\n",
    "   <a href=\"https://www.npmjs.com/package/@julep/sdk\"><img src=\"https://img.shields.io/npm/v/%40julep%2Fsdk?style=social&amp;logo=npm&amp;link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40julep%2Fsdk\" alt=\"NPM Version\" height=\"28\"></a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://pypi.org/project/julep\"><img src=\"https://img.shields.io/pypi/v/julep?style=social&amp;logo=python&amp;label=PyPI&amp;link=https%3A%2F%2Fpypi.org%2Fproject%2Fjulep\" alt=\"PyPI - Version\" height=\"28\"></a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://hub.docker.com/u/julepai\"><img src=\"https://img.shields.io/docker/v/julepai/agents-api?sort=semver&amp;style=social&amp;logo=docker&amp;link=https%3A%2F%2Fhub.docker.com%2Fu%2Fjulepai\" alt=\"Docker Image Version\" height=\"28\"></a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://choosealicense.com/licenses/apache/\"><img src=\"https://img.shields.io/github/license/julep-ai/julep\" alt=\"GitHub License\" height=\"28\"></a>\n",
    "  </p>\n",
    "  \n",
    "  <h3>\n",
    "    <a href=\"https://discord.com/invite/JTSBGRZrzj\" rel=\"dofollow\">Discord</a>\n",
    "    ·\n",
    "    <a href=\"https://x.com/julep_ai\" rel=\"dofollow\">𝕏</a>\n",
    "    ·\n",
    "    <a href=\"https://www.linkedin.com/company/julep-ai\" rel=\"dofollow\">LinkedIn</a>\n",
    "  </h3>\n",
    "</div>\n",
    "\n",
    "## Task Definition: Video Processing with Natural Language\n",
    "\n",
    "### Overview\n",
    "\n",
    "This task leverages the cloudinary `integration` tool, and combines it with a prompt step to convert a natural language instructions and apply them to a given video.\n",
    "\n",
    "### Task Tools:\n",
    "\n",
    "**Cloudinary**: An `integration` type tool that can upload and transform media files.\n",
    "\n",
    "### Task Input:\n",
    "\n",
    "**video_url**: The URL of the video to transform.\n",
    "\n",
    "**public_id**: The public id of the video to transform.\n",
    "\n",
    "**transformation_prompt**: The natural language instructions to apply to the video.\n",
    "\n",
    "### Task Output:\n",
    "\n",
    "**transformed_video_url**: The URL of the transformed video.\n",
    "\n",
    "### Task Flow\n",
    "\n",
    "1. **Input**: The user provides a URL to a video and transformation instructions (in natural language) to apply to the video.\n",
    "\n",
    "2. **Cloudinary Tool Integration**: The `cloudinary_upload` tool is called to upload the video to cloudinary.\n",
    "\n",
    "3. **Prompt Step**: The prompt step is used to convert the natural language instructions into a json of transformation instructions that are compatible with Cloudinary's API. In this step, `gemini-1.5-pro` is used as the model due to its ability to read video files.\n",
    "\n",
    "4. **Cloudinary Tool Integration**: The `cloudinary_upload` tool is called again to apply the transformation instructions to the video.\n",
    "\n",
    "5. **Output**: The final output is the URL of the transformed video.\n",
    "\n",
    "```plaintext\n",
    "+----------+     +------------+     +------------+     +-----------+\n",
    "|  Video   |     | Cloudinary |     |  Prompt    |     | Processed |\n",
    "|  Input   | --> |  Upload    | --> |  Step      | --> |  Video    |\n",
    "|          |     |            |     | (Gemini)   |     |           |\n",
    "+----------+     +------------+     +------------+     +-----------+\n",
    "      |                |                  |                  |\n",
    "      |                |                  |                  |\n",
    "      v                v                  v                  v\n",
    "\"video.mp4 +    Upload video    Generate JSON     \"video.mp4 with\n",
    "NL prompt\"                    transformations     blur & overlay\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "To recreate the notebook and see the code implementation for this task, you can access the Google Colab notebook using the link below:\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/julep-ai/julep/blob/dev/cookbooks/05-video-processing-with-natural-language.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "### Additional Information\n",
    "\n",
    "For more details about the task or if you have any questions, please don't hesitate to contact the author:\n",
    "\n",
    "**Author:** Julep AI  \n",
    "**Contact:** [hey@julep.ai](mailto:hey@julep.ai) or  <a href=\"https://discord.com/invite/JTSBGRZrzj\" rel=\"dofollow\">Discord</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installing the Julep Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade julep --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NOTE:\n",
    "\n",
    "- UUIDs are generated for both the agent and task to uniquely identify them within the system.\n",
    "- Once created, these UUIDs should remain unchanged for simplicity.\n",
    "- Altering a UUID will result in the system treating it as a new agent or task.\n",
    "- If a UUID is changed, the original agent or task will continue to exist in the system alongside the new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "# NOTE: these UUIDs are used in order not to use the `create_or_update` methods instead of\n",
    "# the `create` methods for the sake of not creating new resources every time a cell is run.\n",
    "AGENT_UUID = uuid.uuid4()\n",
    "TASK_UUID = uuid.uuid4()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Julep Client with the API Key\n",
    "\n",
    "Get your API key from [here](https://dashboard.julep.ai/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from julep import Client\n",
    "\n",
    "JULEP_API_KEY = \"YOUR_JULEP_API_KEY\"\n",
    "\n",
    "# Create a Julep client\n",
    "client = Client(api_key=JULEP_API_KEY, environment=\"production\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an \"agent\"\n",
    "\n",
    "Agent is the object to which LLM settings, like model, temperature along with tools are scoped to.\n",
    "\n",
    "To learn more about the agent, please refer to the Agent section in [Julep Concepts](https://docs.julep.ai/docs/concepts/agents)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create agent\n",
    "agent = client.agents.create_or_update(\n",
    "    agent_id=AGENT_UUID,\n",
    "    name=\"Spiderman\",\n",
    "    about=\"AI that can crawl the web and extract data\",\n",
    "    model=\"gpt-4o\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a Task\n",
    "\n",
    "Tasks in Julep are Github-Actions-style workflows that define long-running, multi-step actions.\n",
    "\n",
    "You can use them to conduct complex actions by defining them step-by-step.\n",
    "\n",
    "To learn more about tasks, please refer to the `Tasks` section in [Julep Concepts](https://docs.julep.ai/docs/concepts/tasks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "cloudinary_api_key = \"YOUR_CLOUDINARY_API_KEY\"\n",
    "cloudinary_api_secret = \"YOUR_CLOUDINARY_API_SECRET\"\n",
    "cloudinary_cloud_name = \"YOUR_CLOUDINARY_CLOUD_NAME\"\n",
    "\n",
    "# Define the task\n",
    "task_def = yaml.safe_load(f\"\"\"\n",
    "# yaml-language-server: $schema=https://raw.githubusercontent.com/julep-ai/julep/refs/heads/dev/schemas/create_task_request.json\n",
    "name: Julep Video Processing With Natural Language\n",
    "description: Process a video using natural language instructions\n",
    "\n",
    "########################################################\n",
    "####################### INPUT SCHEMA ###################\n",
    "########################################################\n",
    "input_schema:\n",
    "  type: object\n",
    "  properties:\n",
    "    video_url:\n",
    "      type: string\n",
    "      description: The url of the file to upload\n",
    "    public_id:\n",
    "      type: string\n",
    "      description: The public id of the file to upload\n",
    "    transformation_prompt:\n",
    "      type: string\n",
    "      description: The prompt for the transformations to apply to the file\n",
    "\n",
    "########################################################\n",
    "####################### TOOLS ##########################\n",
    "########################################################\n",
    "                          \n",
    "# Define the tools that the task will use in this workflow\n",
    "tools:\n",
    "- name: cloudinary_upload\n",
    "  type: integration\n",
    "  integration:\n",
    "    provider: cloudinary\n",
    "    method: media_upload\n",
    "    setup:\n",
    "      cloudinary_api_key: \"{cloudinary_api_key}\"\n",
    "      cloudinary_api_secret: \"{cloudinary_api_secret}\"\n",
    "      cloudinary_cloud_name: \"{cloudinary_cloud_name}\"\n",
    "\n",
    "########################################################\n",
    "####################### MAIN WORKFLOW ##################\n",
    "########################################################\n",
    "\n",
    "main:\n",
    "# Step 0: Upload the video to cloudinary\n",
    "- tool: cloudinary_upload\n",
    "  arguments:\n",
    "    file: $ steps[0].input.video_url\n",
    "    public_id: $ steps[0].input.public_id\n",
    "    upload_params:\n",
    "      resource_type: video\n",
    "\n",
    "# Step 1: Generate the transformation json\n",
    "- prompt:\n",
    "  - role: user\n",
    "    content:\n",
    "\n",
    "      - type: text\n",
    "        text: |-\n",
    "          You are a Cloudinary expert. You are given a medial url. it might be an image or a video.\n",
    "          You need to come up with a json of transformations to apply to the given media.\n",
    "          Overall the json could have multiple transformation json objects.\n",
    "          Each transformation json object can have the multiple key value pairs.\n",
    "          Each key value pair should have the key as the transformation name like \"aspect_ratio\", \"crop\", \"width\" etc and the value as the transformation parameter value.\n",
    "          Given below is an example of a transformation json list. Don't provide explanations and/or comments in the json.\n",
    "          ```json\n",
    "          [\n",
    "            {{\n",
    "              \"aspect_ratio\": \"1.0\",\n",
    "              \"width\": 250,\n",
    "            }},\n",
    "            {{\n",
    "              \"fetch_format\": \"auto\"\n",
    "            }},\n",
    "            {{\n",
    "              \"overlay\":\n",
    "              {{\n",
    "                \"url\": \"<image_url>\"\n",
    "              }}\n",
    "            }},\n",
    "            {{\n",
    "              \"flags\": \"layer_apply\"\n",
    "            }}\n",
    "          ]\n",
    "          ```\n",
    "      - type: image_url\n",
    "        image_url:\n",
    "          url: $ _.url\n",
    "\n",
    "      - type: text\n",
    "        text: |-\n",
    "          $ f'''Hey, check the video above, I need to apply the following transformations using cloudinary.\n",
    "          {{steps[0].input.transformation_prompt}}'''\n",
    "\n",
    "  unwrap: true\n",
    "  settings:\n",
    "    model: gemini/gemini-1.5-pro\n",
    "\n",
    "# Step 2: Extract the json from the model's response\n",
    "- evaluate:\n",
    "    model_transformation: >-\n",
    "      $ load_json(\n",
    "        _[_.find(\"```json\")+7:][:_[_.find(\"```json\")+7:].find(\"```\")])\n",
    "\n",
    "# Step 3: Upload the video to cloudinary\n",
    "- tool: cloudinary_upload\n",
    "  arguments:\n",
    "    file: $ steps[0].input.video_url\n",
    "    public_id: $ steps[0].input.public_id\n",
    "    upload_params:\n",
    "      transformation: $ _.model_transformation\n",
    "      resource_type: video\n",
    "\n",
    "# Step 4: Return the transformed video url\n",
    "- evaluate:\n",
    "    transformed_video_url: $ _.url\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:olive;\">Notes:</span>\n",
    "- The `unwrap: True` in the prompt step is used to unwrap the output of the prompt step (to unwrap the `choices[0].message.content` from the output of the model).\n",
    "- The `$` sign is used to differentiate between a Python expression and a string.\n",
    "- The `_` refers to the output of the previous step.\n",
    "- The `steps[index].input` refers to the input of the step at `index`.\n",
    "- The `steps[index].output` refers to the output of the step at `index`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating the task object\n",
    "task = client.tasks.create_or_update(\n",
    "    task_id=TASK_UUID,\n",
    "    agent_id=AGENT_UUID,\n",
    "    **task_def\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an Execution\n",
    "\n",
    "An execution is a single run of a task. It is a way to run a task with a specific set of inputs.\n",
    "\n",
    "To learn more about executions, please refer to the `Executions` section in [Julep Concepts](https://docs.julep.ai/docs/concepts/execution)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'067a60aa-a389-7e5a-8000-1f47c51b0ced'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating an execution object\n",
    "transformation_prompt = \"\"\"\n",
    "1- I want to add an overlay an the following image to the video, and apply a layer apply flag also. Here's the image url:\n",
    "https://res.cloudinary.com/demo/image/upload/logos/cloudinary_icon_white.png\n",
    "\n",
    "2- I also want you to to blur the video, and add a fade in and fade out effect to the video with a duration of 3 seconds each.\n",
    "\"\"\"\n",
    "\n",
    "input_video_url = \"http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4\"\n",
    "\n",
    "execution = client.executions.create(\n",
    "    task_id=TASK_UUID,\n",
    "    input={\n",
    "        \"video_url\":  input_video_url,\n",
    "        \"public_id\": \"video_test2\",\n",
    "        \"transformation_prompt\": transformation_prompt,\n",
    "    }\n",
    ")\n",
    "execution.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking execution details and output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to get the execution details and the output:\n",
    "\n",
    "1. **Get Execution Details**: This method retrieves the details of the execution, including the output of the last transition that took place.\n",
    "\n",
    "2. **List Transitions**: This method lists all the task steps that have been executed up to this point in time, so the output of a successful execution will be the output of the last transition (first in the transition list as it is in reverse chronological order), which should have a type of `finish`.\n",
    "\n",
    "\n",
    "<span style=\"color:olive;\">Note: You need to wait for a few seconds for the execution to complete before you can get the final output, so feel free to run the following cells multiple times until you get the final output.</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'public_id': 'video_test2', 'video_url': 'http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4', 'transformation_prompt': \"\\n1- I want to add an overlay an the following image to the video, and apply a layer apply flag also. Here's the image url:\\nhttps://res.cloudinary.com/demo/image/upload/logos/cloudinary_icon_white.png\\n\\n2- I also want you to to blur the video, and add a fade in and fade out effect to the video with a duration of 3 seconds each.\\n\"}\n"
     ]
    }
   ],
   "source": [
    "# Get execution details\n",
    "execution = client.executions.get(execution.id)\n",
    "# Print the output\n",
    "print(execution.output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transition type:  init\n",
      "Transition output:  {'public_id': 'video_test2', 'video_url': 'http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4', 'transformation_prompt': \"\\n1- I want to add an overlay an the following image to the video, and apply a layer apply flag also. Here's the image url:\\nhttps://res.cloudinary.com/demo/image/upload/logos/cloudinary_icon_white.png\\n\\n2- I also want you to to blur the video, and add a fade in and fade out effect to the video with a duration of 3 seconds each.\\n\"}\n",
      "--------------------------------------------------\n",
      "Transition type:  step\n",
      "Transition output:  {'url': 'https://res.cloudinary.com/dpnjjk8mb/video/upload/v1738934100/video_test2.mp4', 'base64': None, 'meta_data': {'url': 'http://res.cloudinary.com/dpnjjk8mb/video/upload/v1738934100/video_test2.mp4', 'tags': [], 'type': 'upload', 'audio': {'codec': 'aac', 'bit_rate': '191999', 'channels': 2, 'frequency': 44100, 'channel_layout': 'stereo'}, 'bytes': 2252313, 'video': {'codec': 'h264', 'level': 31, 'profile': 'High', 'bit_rate': '1002377', 'time_base': '1/48', 'pix_format': 'yuv420p'}, 'width': 1280, 'format': 'mp4', 'height': 720, 'api_key': '518455844981529', 'version': 1738934100, 'asset_id': '069d70c84e88ee9293a1d753b3ca3898', 'bit_rate': 1197518, 'duration': 15.046531, 'existing': True, 'is_audio': False, 'rotation': 0, 'nb_frames': 361, 'signature': '836eddb515d796dc8484d7dc108cb67fbe0ba1d5', 'created_at': '2024-11-20T08:55:44Z', 'frame_rate': 24.0, 'version_id': '85846ee1c860645e1149e8029051c07d', 'placeholder': False, 'asset_folder': '', 'display_name': 'video_test2', 'playback_url': 'https://res.cloudinary.com/dpnjjk8mb/video/upload/sp_auto/v1738934100/video_test2.m3u8', 'resource_type': 'video'}, 'public_id': 'video_test2'}\n",
      "--------------------------------------------------\n",
      "Transition type:  step\n",
      "Transition output:  ```json\n",
      "[\n",
      "  {\n",
      "    \"overlay\": {\n",
      "      \"url\": \"https://res.cloudinary.com/demo/image/upload/logos/cloudinary_icon_white.png\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"flags\": \"layer_apply\"\n",
      "  },\n",
      "  {\n",
      "    \"effect\": \"blur:200\"\n",
      "  },\n",
      "  {\n",
      "    \"effect\": \"fadeIn:3000\"\n",
      "  },\n",
      "  {\n",
      "    \"effect\": \"fadeOut:3000\"\n",
      "  }\n",
      "]\n",
      "```\n",
      "--------------------------------------------------\n",
      "Transition type:  step\n",
      "Transition output:  {'model_transformation': [{'overlay': {'url': 'https://res.cloudinary.com/demo/image/upload/logos/cloudinary_icon_white.png'}}, {'flags': 'layer_apply'}, {'effect': 'blur:200'}, {'effect': 'fadeIn:3000'}, {'effect': 'fadeOut:3000'}]}\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Lists all the task steps that have been executed up to this point in time\n",
    "transitions = client.executions.transitions.list(execution_id=execution.id).items\n",
    "\n",
    "# Transitions are retrieved in reverse chronological order\n",
    "for transition in reversed(transitions):\n",
    "    print(\"Transition type: \", transition.type)\n",
    "    print(\"Transition output: \", transition.output)\n",
    "    print(\"-\"*50)\n",
    "\n",
    "if transitions[0].type == \"finish\":\n",
    "    transformed_video_url = transitions[0].output['transformed_video_url']"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}