recipes/use_cases/github_triage/walkthrough.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Automatic Issues Triaging with Llama\n",
        "\n",
        "We utilize an off-the-shelf Llama model to analyze, generate insights, and create a report for better understanding of the state of a repository. \n",
        "\n",
        "This notebook walks you through the tool's working. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!git clone https://github.com/meta-llama/llama-recipes\n",
        "%cd recipes/use_cases/github_triage\n",
        "!pip install -r requirements.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "import yaml\n",
        "import os\n",
        "\n",
        "from utils import fetch_repo_issues, validate_df_values\n",
        "from plots import draw_all_plots\n",
        "from pdf_report import create_report_pdf\n",
        "from triage import generate_executive_reports, generate_issue_annotations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Set access keys and tokens\n",
        "\n",
        "Set your GitHub token for API calls. Some privileged information may not be available if you don't have push-access to the target repository.\n",
        "\n",
        "Set your groq token for inference. Get one at https://console.groq.com/keys"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "github_token = input(\"Enter your Github API token\")\n",
        "groq_token = input(\"Enter your Groq token\")\n",
        "\n",
        "with open(\"config.yaml\", \"r\") as f:\n",
        "  CFG = yaml.safe_load(f)\n",
        "CFG['github_token'] = github_token\n",
        "CFG['model']['groq']['key'] = groq_token\n",
        "with open(\"config.yaml\", \"w\") as f:\n",
        "  yaml.dump(CFG, f)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Set target repo and period to analyze"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Repo name:  pytorch/pytorch\n",
            "Period:  2024-08-24 - 2024-08-29\n"
          ]
        }
      ],
      "source": [
        "repo_name = input(\"What repository do you want to analyze? (eg: meta-llama/llama-recipes)\")\n",
        "start_date = input(\"Start analysis (eg: 2024-08-23): \")\n",
        "end_date = input(\"End analysis (eg: 2024-08-30): \")\n",
        "\n",
        "out_folder = f'output/{repo_name}/{start_date}_{end_date}'\n",
        "os.makedirs(out_folder, exist_ok=True)\n",
        "\n",
        "print(\"Repo name: \", repo_name)\n",
        "print(\"Period: \", start_date, \"-\", end_date)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Fetch issues from the repository\n",
        "\n",
        "Use the github API to retrieve issues (including the full discussion on them) and store it in a dataframe."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Fetching issues on pytorch/pytorch from 2024-08-24 to 2024-08-29\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Fetched 112 issues on pytorch/pytorch from 2024-08-24 to 2024-08-29\n",
            "Data saved to output/pytorch/pytorch/2024-08-24_2024-08-29/issues.csv\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "[Showing 5 out of 112 rows]\n",
            "\n",
            "         repo_name  number                                          html_url  \\\n",
            "0  pytorch/pytorch  134384  https://github.com/pytorch/pytorch/issues/134384   \n",
            "1  pytorch/pytorch  134385  https://github.com/pytorch/pytorch/issues/134385   \n",
            "2  pytorch/pytorch  134388  https://github.com/pytorch/pytorch/issues/134388   \n",
            "3  pytorch/pytorch  134390  https://github.com/pytorch/pytorch/issues/134390   \n",
            "4  pytorch/pytorch  134391  https://github.com/pytorch/pytorch/issues/134391   \n",
            "\n",
            "   closed  num_comments            created_at  \\\n",
            "0   False             3  2024-08-24T00:26:26Z   \n",
            "1   False             2  2024-08-24T00:43:18Z   \n",
            "2   False             2  2024-08-24T05:04:42Z   \n",
            "3   False             2  2024-08-24T07:10:28Z   \n",
            "4    True             1  2024-08-24T07:54:45Z   \n",
            "\n",
            "                                          discussion  \n",
            "0  Torch compile stochastically fails with FileNo...  \n",
            "1  FlopCounterMode doesn't support HOP\\n### 🐛 Des...  \n",
            "2  Remove redundant copy in functional collective...  \n",
            "3  This send_object_list and recv_object_list met...  \n",
            "4  ProcessGroupNCCL::barrier()'s device id guessi...  \n"
          ]
        }
      ],
      "source": [
        "issues_df = fetch_repo_issues(repo_name, start_date, end_date)\n",
        "issues_df = validate_df_values(issues_df, out_folder, 'issues')\n",
        "\n",
        "print(f\"\\n\\n[Showing 5 out of {issues_df.shape[0]} rows]\\n\")\n",
        "print(issues_df.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Use Llama to generate the annotations for this data\n",
        "\n",
        "We use 2 prompts defined in `config.yaml` to annotate the issues with additional information that can help triagers and repo maintainers:\n",
        "1. `parse_issues`: generate annotations and other metadata basd on the contents in the issue thread.\n",
        "   \n",
        "2. `assign_category` tags each issue with the most relevant category (from a list of categories specified in the prompt's output schema)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Prompt for generating annotations:\n",
            "You are an expert maintainer of an open source project. Given some discussion threads, you must respond with a report in JSON. Your response should only contain English, and you may translate if you can.\n",
            "\n",
            "Prompt for categorizing issues:\n",
            "You are the lead maintainer of an open source project. Given a list of issues, generate a JSON that categorizes the issues by common themes. For every theme include a description and cite the relevant issue numbers. All issues must be categorized into at least one theme.\n",
            "\n",
            "Issues can be categorized into: ['Cloud Compute', 'Installation and Environment', 'Model Loading', 'Model Fine-tuning and Training', 'Model Conversion', 'Model Inference', 'Distributed Training and Multi-GPU', 'Performance and Optimization', 'Quantization and Mixed Precision', 'Documentation', 'CUDA Compatibility', 'Model Evaluation and Benchmarking', 'Miscellaneous', 'Invalid']\n"
          ]
        }
      ],
      "source": [
        "print(f\"Prompt for generating annotations:\\n{CFG['prompts']['parse_issue']['system']}\\n\")\n",
        "print(f\"Prompt for categorizing issues:\\n{CFG['prompts']['assign_category']['system']}\\n\")\n",
        "print(f\"Issues can be categorized into: {eval(CFG['prompts']['assign_category']['json_schema'])['properties']['report']['items']['properties']['theme']['enum']}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We run inference on these prompts along with the issues data in `issues_df`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Generating annotations for 112\n",
            "Data saved to output/pytorch/pytorch/2024-08-24_2024-08-29/annotated_issues.csv\n",
            "Data saved to output/pytorch/pytorch/2024-08-24_2024-08-29/annotated_issues.csv\n"
          ]
        }
      ],
      "source": [
        "# Generate annotations and metadata\n",
        "annotated_issues, theme_counts = generate_issue_annotations(issues_df)\n",
        "\n",
        "# Validate and save generated data\n",
        "annotated_issues = validate_df_values(annotated_issues, out_folder, 'annotated_issues')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "* The annotations include new metadata like `summary`, `possible_causes`, `remediations` that can help triagers quickly understand and diagnose the issue. \n",
        "\n",
        "* Annotations like `issue_type`, `component`, `themes` can help identify the right POC / maintainer to address the issue.\n",
        "\n",
        "* Annotations like `severity`, `op_expertise` and `sentiment` can help gauge the general pulse of developers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "metadata generated by LLM: {'possible_causes', 'summary', 'severity', 'issue_type', 'sentiment', 'themes', 'component', 'op_expertise', 'remediations'}\n",
            "\n",
            "\n",
            "[Showing 5 out of 112 rows]\n",
            "\n",
            "         repo_name  number                                          html_url  \\\n",
            "0  pytorch/pytorch  134384  https://github.com/pytorch/pytorch/issues/134384   \n",
            "1  pytorch/pytorch  134385  https://github.com/pytorch/pytorch/issues/134385   \n",
            "2  pytorch/pytorch  134388  https://github.com/pytorch/pytorch/issues/134388   \n",
            "3  pytorch/pytorch  134390  https://github.com/pytorch/pytorch/issues/134390   \n",
            "4  pytorch/pytorch  134391  https://github.com/pytorch/pytorch/issues/134391   \n",
            "\n",
            "   closed  num_comments            created_at  \\\n",
            "0   False             3  2024-08-24T00:26:26Z   \n",
            "1   False             2  2024-08-24T00:43:18Z   \n",
            "2   False             2  2024-08-24T05:04:42Z   \n",
            "3   False             2  2024-08-24T07:10:28Z   \n",
            "4    True             1  2024-08-24T07:54:45Z   \n",
            "\n",
            "                                          discussion  \\\n",
            "0  Torch compile stochastically fails with FileNo...   \n",
            "1  FlopCounterMode doesn't support HOP\\n### 🐛 Des...   \n",
            "2  Remove redundant copy in functional collective...   \n",
            "3  This send_object_list and recv_object_list met...   \n",
            "4  ProcessGroupNCCL::barrier()'s device id guessi...   \n",
            "\n",
            "                                             summary  \\\n",
            "0  Torch compile stochastically fails with FileNo...   \n",
            "1  FlopCounterMode does not support HOP, causing ...   \n",
            "2  Discussion on removing redundant copy operatio...   \n",
            "3  Performance issue with send_object_list and re...   \n",
            "4  ProcessGroupNCCL::barrier() device id guessing...   \n",
            "\n",
            "                                     possible_causes  \\\n",
            "0  [Race condition due to concurrent.futures, Fil...   \n",
            "1  [Missing registration of FlopCounterMode formu...   \n",
            "2  [Current implementation of functional collecti...   \n",
            "3  [Serialization and deserialization overhead, D...   \n",
            "4  [Insufficient device id guessing logic in Proc...   \n",
            "\n",
            "                                        remediations  \\\n",
            "0  [Check for file existence before compiling, Us...   \n",
            "1  [Register a FlopCounterMode formula via HOP.py...   \n",
            "2  [Implement in-place version of functional coll...   \n",
            "3  [Use send/recv methods instead of send_object_...   \n",
            "4  [Improve the device id guessing logic in Proce...   \n",
            "\n",
            "                                  component sentiment  issue_type severity  \\\n",
            "0                             torch._dynamo  negative  bug_report    major   \n",
            "1  torch.utils.flop_counter.FlopCounterMode   neutral  bug_report    major   \n",
            "2                       PyTorch Distributed   neutral  discussion    minor   \n",
            "3         PyTorch NCCL communication module  negative  bug_report    major   \n",
            "4                          ProcessGroupNCCL  negative  bug_report    major   \n",
            "\n",
            "   op_expertise                                             themes  \n",
            "0      advanced  [Distributed Training and Multi-GPU, CUDA Comp...  \n",
            "1      advanced                                    [Miscellaneous]  \n",
            "2      advanced                                    [Miscellaneous]  \n",
            "3  intermediate  [Distributed Training and Multi-GPU, Performan...  \n",
            "4      advanced  [Distributed Training and Multi-GPU, Performan...  \n"
          ]
        }
      ],
      "source": [
        "\n",
        "print(f\"Additional metadata generated by LLM: {set(annotated_issues.columns).difference(set(issues_df.columns))}\\n\\n\")\n",
        "print(f\"[Showing 5 out of {annotated_issues.shape[0]} rows]\\n\")\n",
        "print(annotated_issues.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Use Llama to generate high-level insights\n",
        "\n",
        "The above data is good for OSS maintainers and developers to quickly address any issues. The next section will synthesize this data into high-level insights about this repository."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Prompt for generating high-level overview:\n",
            "\n",
            "{'system': 'You are not only an experienced Open Source maintainer, but also an expert at paraphrasing raw data into clear succinct reports. Draft a concise report about the issues in this open source repository. Include an executive summary that provides an overview of the challenges faced, any open questions or decisions to be made, or actions that we can take. Group issues together if they ladder up to the same overall challenge, summarize the challenges and include any actionable resolutions we can take (more information in the \\\\\"remediations\\\\\" sections). Use your experience and judgement to ignore issues that are clearly unrelated to the open source project. Ensure the output is in JSON.', 'json_schema': '{ \"type\": \"object\", \"properties\": { \"executive_summary\": { \"description\": \"An executive summary of the analysis\", \"type\": \"string\" }, \"open_questions\": { \"description\": \"Any open questions or decisions that the product team needs to make in light of these issues\", \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"issue_analysis\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"properties\": { \"key_challenge\": { \"description\": \"A description of the challenge reported in these issues\", \"type\": \"string\" }, \"affected_issues\": { \"description\": \"A list of issues that are related to this challenge\", \"type\": \"array\", \"items\": { \"type\": \"number\" } }, \"possible_causes\": { \"description\": \"A list of possible causes or reasons for this challenge to occur\", \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"remediations\": { \"description\": \"Steps we can take to address this challenge\", \"type\": \"array\", \"items\": { \"type\": \"string\" } } }, \"required\": [\"key_challenge\", \"affected_issues\", \"possible_causes\", \"remediations\"] } } }, \"required\": [\"issue_analysis\", \"open_questions\", \"actions\", \"executive_summary\"] }'}\n"
          ]
        }
      ],
      "source": [
        "print(f\"Prompt for generating high-level overview:\\n\\n{CFG['prompts']['get_overview']}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Generating high-level summaries from annotations...\n",
            "Identifying key-challenges faced by users...\n",
            "Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/challenges.csv\n",
            "Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/overview.csv\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "# Generate high-level analysis\n",
        "challenges, overview = generate_executive_reports(annotated_issues, theme_counts, repo_name, start_date, end_date)\n",
        "# Validate and save generated data\n",
        "challenges = validate_df_values(challenges, out_folder, 'challenges')\n",
        "overview = validate_df_values(overview, out_folder, 'overview')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Key Challenges data\n",
        "\n",
        "We identify key areas that users are challenged by along with the relevant issues."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[Showing 5 out of 5 rows]\n",
            "\n",
            "         repo_name  start_date    end_date                    key_challenge  \\\n",
            "0  pytorch/pytorch  2024-08-28  2024-08-28          Performance Regressions   \n",
            "1  pytorch/pytorch  2024-08-28  2024-08-28             Compatibility Issues   \n",
            "2  pytorch/pytorch  2024-08-28  2024-08-28         Security Vulnerabilities   \n",
            "3  pytorch/pytorch  2024-08-28  2024-08-28  Tensor Parallelism and Autograd   \n",
            "4  pytorch/pytorch  2024-08-28  2024-08-28                     CUDA Support   \n",
            "\n",
            "            affected_issues  \\\n",
            "0          [134679, 134686]   \n",
            "1  [134640, 134682, 134684]   \n",
            "2                  [134664]   \n",
            "3          [134668, 134676]   \n",
            "4          [134682, 134684]   \n",
            "\n",
            "                                     possible_causes  \\\n",
            "0  [Changes in the torch or torchvision libraries...   \n",
            "1  [Incompatible version of caffe2 with the curre...   \n",
            "2  [Using the affected version of protobuf (3.20....   \n",
            "3  [Lack of understanding of tensor parallelism i...   \n",
            "4  [cuDNN version incompatibility between PyTorch...   \n",
            "\n",
            "                                        remediations  \n",
            "0  [Investigate the suspected guilty commit and r...  \n",
            "1  [Try installing an older version of caffe2 tha...  \n",
            "2  [Update protobuf to a version mentioned in the...  \n",
            "3  [Improve documentation on tensor parallelism a...  \n",
            "4  [Ensure PyTorch can find the bundled cuDNN by ...  \n"
          ]
        }
      ],
      "source": [
        "print(f\"[Showing 5 out of {challenges.shape[0]} rows]\\n\")\n",
        "print(challenges.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Overview Data\n",
        "\n",
        "As the name suggests, the `overview` dataframe contains columns that provide information about the overall activity in the repository during this period, including:\n",
        "* an executive summary of all the issues seen during this period\n",
        "* how many issues were created, discussed and closed\n",
        "* what are some open questions that the maintainers should address\n",
        "* how many issues were seen for each theme etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[Showing 5 out of 1 rows]\n",
            "\n",
            "         repo_name  start_date    end_date  issues_created  open_discussion  \\\n",
            "0  pytorch/pytorch  2024-08-28  2024-08-28              26                8   \n",
            "\n",
            "   closed_discussion  open_no_discussion  closed_no_discussion  \\\n",
            "0                  0                  18                     0   \n",
            "\n",
            "                                      open_questions  \\\n",
            "0  [What is the root cause of the performance reg...   \n",
            "\n",
            "                                   executive_summary  ...  \\\n",
            "0  The PyTorch repository is facing various chall...  ...   \n",
            "\n",
            "   themes_count_model_loading  themes_count_model_fine_tuning_and_training  \\\n",
            "0                           3                                            3   \n",
            "\n",
            "   themes_count_model_inference  \\\n",
            "0                             3   \n",
            "\n",
            "   themes_count_distributed_training_and_multi_gpu  \\\n",
            "0                                                3   \n",
            "\n",
            "   themes_count_performance_and_optimization  \\\n",
            "0                                          5   \n",
            "\n",
            "   themes_count_quantization_and_mixed_precision  themes_count_documentation  \\\n",
            "0                                              1                           1   \n",
            "\n",
            "   themes_count_cuda_compatibility  \\\n",
            "0                                2   \n",
            "\n",
            "   themes_count_model_evaluation_and_benchmarking  themes_count_miscellaneous  \n",
            "0                                               1                          14  \n",
            "\n",
            "[1 rows x 28 columns]\n"
          ]
        }
      ],
      "source": [
        "print(f\"[Showing 5 out of {overview.shape[0]} rows]\\n\")\n",
        "print(overview.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Visualizing the data\n",
        "\n",
        "Based on this data we can easily create some plots to graphically understand the activity in the repo.\n",
        "\n",
        "Some additional data can be accessed via the github API, but this requires you to have push-access to this repo.\n",
        "\n",
        "The generated plots are saved as images in `plot_folder`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Plotting traffic trends...\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Github fetch failed for <function plot_views_clones at 0x13837a3e0>. Make sure you have push-access to pytorch/pytorch!\n",
            "Github fetch failed for <function plot_high_traffic_resources at 0x13ed59580>. Make sure you have push-access to pytorch/pytorch!\n",
            "Github fetch failed for <function plot_high_traffic_referrers at 0x13ed84860>. Make sure you have push-access to pytorch/pytorch!\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Plotting issue trends...\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "# Create graphs and charts\n",
        "plot_folder = out_folder + \"/plots\"\n",
        "os.makedirs(plot_folder, exist_ok=True)\n",
        "draw_all_plots(repo_name, plot_folder, overview)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Putting it together\n",
        "\n",
        "Now that we have all the data and insights, we can create a PDF report"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Creating PDF report at output/pytorch/pytorch/2024-08-28_2024-08-28/report.pdf\n"
          ]
        }
      ],
      "source": [
        "exec_summary = overview['executive_summary'].iloc[0]\n",
        "open_qs = overview['open_questions'].iloc[0]\n",
        "key_challenges_data = challenges[['key_challenge', 'possible_causes', 'remediations', 'affected_issues']].to_dict('records')\n",
        "create_report_pdf(repo_name, start_date, end_date, key_challenges_data, exec_summary, open_qs, out_folder)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from IPython.display import IFrame\n",
        "IFrame(\"output/pytorch/pytorch/2024-08-28_2024-08-28/report.pdf\", width=900, height=800)"
      ]
    }
  ],
  "metadata": {
    "fileHeader": "",
    "fileUid": "4f3f888b-4ccb-4206-8b8c-9898409f09ed",
    "isAdHoc": false,
    "kernelspec": {
      "display_name": "haystack",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.5"
    }
  }
}