Colab-Solutions.ipynb

{
  "metadata": {
    "colab": {
      "name": "pydata-llama-index-tutorial-colab.ipynb",
      "private_outputs": false,
      "provenance": [],
      "cell_execution_strategy": "setup"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "id": "12976e25-8fc1-48d4-9186-9d2bcea0de92",
      "metadata": {},
      "source": [
        "# Running this notebook in Google Colab\n",
        "This notebook is intended to run in Google Colab [here](https://colab.research.google.com/github/datakami/pydata-llama-index-tutorial/blob/main/Colab-Exercises.ipynb).\n",
        "\n",
        "Simply execute the code below to install the neccessary dependencies and download the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9da5883c-d4ec-43e0-bc95-835f875b6c77",
      "metadata": {},
      "outputs": [],
      "source": [
        "%pip install llama-index==0.8.5.post2 sentence-transformers==2.2.2 loguru==0.7.0\n",
        "# download the data\n",
        "from urllib.request import urlretrieve\n",
        "path, _ = urlretrieve(\"https://github.com/datakami/pydata-llama-index-tutorial/archive/refs/heads/main.zip\")\n",
        "import zipfile\n",
        "with zipfile.ZipFile(path, 'r') as zip_ref:\n",
        "    zip_ref.extractall(\".\")\n",
        "!mv pydata-llama-index-tutorial-main/data pydata-llama-index-tutorial-main/indices ."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c913d2a2-f87f-4505-abb4-49e6b2291e52",
      "metadata": {},
      "source": [
        "# Part 1: Querying"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0e084dce-bd60-43c1-a13d-cd9636ea2856",
      "metadata": {},
      "source": [
        "If you want to use the OpenAPI API during this tutorial, make a file `secret.py` with `openai_api_key = \"YOURKEYHERE\"`.\n",
        "\n",
        "How to use this notebook: \n",
        "1. Execute all cells under \"Setup\"\n",
        "2. Fill in the exercises below"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8feb43be-6be1-4cd1-933f-cd3a77052e33",
      "metadata": {},
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cdb21f09-827a-412c-8b19-c40fd14825a1",
      "metadata": {},
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "effd39ab-b6ba-4d69-8e8f-39f382d7712f",
      "metadata": {},
      "outputs": [],
      "source": [
        "import pprint\n",
        "import os\n",
        "import sys\n",
        "from pathlib import Path\n",
        "\n",
        "# logging for lazy people :)\n",
        "from loguru import logger\n",
        "\n",
        "# we're not importing specific methods or classes so it's clear when we actually call llama_index!\n",
        "import llama_index"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4b056809-10bc-4f4d-90bf-a9b8e8acd34e",
      "metadata": {},
      "source": [
        "### Logging"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5f44daa1-a3e6-45bf-bc6e-8c84fad646b2",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "logger.remove()\n",
        "logger.add(sys.stdout, format=\"{time} - {level} - {message}\", level=\"DEBUG\")\n",
        "logger.add(\"tutorial_part_1.log\", level=\"DEBUG\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ee221890-7faa-4b1a-b593-ffe15d5976dc",
      "metadata": {},
      "source": [
        "### Tell the notebook where files are stored"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "84e2e8b1-5b94-4dcf-b109-9e8351b087b4",
      "metadata": {},
      "outputs": [],
      "source": [
        "DATA_PATH = Path(\"data/pydata/schedule.json\")\n",
        "INDEX_PATH = Path(\"indices/pydata_schedule_index/\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6b4f10b1-759c-459b-bece-1495c099cf62",
      "metadata": {},
      "source": [
        "### Tell `llama-index` to use a local embeddings model for retrieval"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3420c723-b6f0-4c1f-8a5f-9da6b875747b",
      "metadata": {},
      "source": [
        "More information about this embeddings model: https://www.sbert.net/docs/pretrained_models.html#model-overview\n",
        "\n",
        "Note that all-minilm-l6-v2 has a maximum size of 256 tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0c4b220f-e86c-48e6-a069-ec4d84a2b119",
      "metadata": {},
      "outputs": [],
      "source": [
        "embed_model = \"local:sentence-transformers/all-minilm-l6-v2\"\n",
        "llm = None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85531cae-9ec3-47f3-b4c1-e9df4cea39f4",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "LLM is explicitly disabled. Using MockLLM.\n"
          ]
        }
      ],
      "source": [
        "service_context = llama_index.ServiceContext.from_defaults(\n",
        "  embed_model=embed_model, chunk_size=256, llm=llm\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7b317df6-0c8c-4ec8-beb9-ce932bc34a50",
      "metadata": {},
      "source": [
        "### Load a vector index with the PyData Amsterdam 2023 schedule"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "81545f8f-f540-4b5a-bc39-eac70da0771a",
      "metadata": {},
      "source": [
        "Load the index from file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3b8bc3e5-6bc7-4970-9a1f-c722ba931956",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-09-04T10:54:01.995195+0200 - INFO - Loaded index from local storage\n"
          ]
        }
      ],
      "source": [
        "if not os.path.exists(INDEX_PATH):\n",
        "    logger.error(\"Index file for part 1 does not exist on disk. :(\")\n",
        "else:\n",
        "    try:                                                                             \n",
        "        storage_context = llama_index.StorageContext.from_defaults(persist_dir=INDEX_PATH)\n",
        "        index = llama_index.load_index_from_storage(storage_context, service_context=service_context)\n",
        "        logger.info(\"Loaded index from local storage\")                               \n",
        "    except Exception as e:                                                           \n",
        "        logger.error(e) "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cede9a40-f6e1-42f3-92bf-4519bfad87ef",
      "metadata": {},
      "source": [
        "## Exercises"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4b4d0cad-f3bb-4467-b3ed-ec87e5fa966a",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create a retriever from the vector index `index`\n",
        "retriever = index.as_retriever()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f343d465-8b22-4778-91e7-b33f3c63e826",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Retrieve chunks that mention llama-index\n",
        "results = retriever.retrieve(\"llama_index\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6741e34d-0edb-4aae-97d6-f1760ba8e3e3",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "node=TextNode(id_='e5163ca0-dbf3-455c-9577-905893dc33a1', embedding=None, metadata={'title': 'Building a personal search engine with llama-index', 'speakers': 'Judith van Stegeren, Yorick van Pelt'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='f71f1eae-c12e-5b4f-8902-b8d0b8a38d72', node_type=None, metadata={'title': 'Building a personal search engine with llama-index', 'speakers': 'Judith van Stegeren, Yorick van Pelt'}, hash='ad9a94ef0dbb13afaa9a4a03c12f20c93873fc7fab5cf57bc88176de147749a0'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='db5d421e-0e1d-4e6f-98b0-a98866ff9d61', node_type=None, metadata={'title': 'Building a personal search engine with llama-index', 'speakers': 'Judith van Stegeren, Yorick van Pelt'}, hash='2d9407d1e16cbae760a7599b871c7d5e22787d217f522a204e2c310b0d68a0cf')}, hash='e279497d370f982bc12a34c7f0eb19e888975d85b14b5a4c3218d733c2c7d55e', text='For the demo application, we will only use open-source software and models that are light enough to run on an average laptop without a GPU.Using llama-index with OpenAI\u2019s API is optional.If you want to explore postprocessing your results with GPT-3.5, we recommend registering an OpenAI account and making sure you have your OpenAI API key ready.', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n') score=0.5057451278480072\n",
            "\n",
            "node=TextNode(id_='db5d421e-0e1d-4e6f-98b0-a98866ff9d61', embedding=None, metadata={'title': 'Building a personal search engine with llama-index', 'speakers': 'Judith van Stegeren, Yorick van Pelt'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='f71f1eae-c12e-5b4f-8902-b8d0b8a38d72', node_type=None, metadata={'title': 'Building a personal search engine with llama-index', 'speakers': 'Judith van Stegeren, Yorick van Pelt'}, hash='ad9a94ef0dbb13afaa9a4a03c12f20c93873fc7fab5cf57bc88176de147749a0'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='e5163ca0-dbf3-455c-9577-905893dc33a1', node_type=None, metadata={'title': 'Building a personal search engine with llama-index', 'speakers': 'Judith van Stegeren, Yorick van Pelt'}, hash='e279497d370f982bc12a34c7f0eb19e888975d85b14b5a4c3218d733c2c7d55e')}, hash='2d9407d1e16cbae760a7599b871c7d5e22787d217f522a204e2c310b0d68a0cf', text='Building a personal search engine with llama-index\\n\\nWouldn\u2019t it be great to have a Google-like search engine, but then for your own text files and completely private?In this tutorial we\u2019ll build a small personal search engine using open source library llama-index.In this tutorial we will build a small personal search engine using open source library `llama-index`.Llama-index provides utility functions for ingesting various kinds of data, breaking the data up in chunks, building an index of that data using vector embeddings, and retrieving data from the index based on queries.We can even use llama-index to post-process the retrieval results for us using large language models such as GPT.The target audience is people that are already familiar with Python.Participants will experience working with unstructured data, vector embeddings, and explore the possibilities of the recent developments in natural language processing.Workshop materials will be provided via Github before the start of the workshop.', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n') score=0.4092205934719865\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for result in results:\n",
        "    print(result)\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "01462a67-ebcc-4282-a7c0-5fadb0cc75a9",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "node=TextNode(id_='65b097e3-7cab-42d1-85b6-a850b1a4d33c', embedding=None, metadata={'title': 'The proof of the pudding is in the (way of) eating: quasi-experimental methods of causal inference and their practical pitfalls', 'speakers': 'Jakob Willisch'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e5be9c09-4c7c-5a3f-b1f8-2c7ecec74cd8', node_type=None, metadata={'title': 'The proof of the pudding is in the (way of) eating: quasi-experimental methods of causal inference and their practical pitfalls', 'speakers': 'Jakob Willisch'}, hash='7ea1436f06df89779e327b88d70a9064aeb40f842a522e502f74727cf3287d1a'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='a3b902b3-1144-44b8-9ae9-0b53e7029138', node_type=None, metadata={'title': 'The proof of the pudding is in the (way of) eating: quasi-experimental methods of causal inference and their practical pitfalls', 'speakers': 'Jakob Willisch'}, hash='bf37f1115e40ebf303a5d3beb1285a35825f84695ee5832141c289aa6caa52be')}, hash='fd918f91548e8d09c08b03d6358d92706ccd3e44e9f1f3ffd95d5bcc8df43388', text=\"The proof of the pudding is in the (way of) eating: quasi-experimental methods of causal inference and their practical pitfalls\\n\\nData scientists and analysts are using quasi-experimental methods to make recommendations based on causality instead of randomized control trials.While these methods are easy to use, their assumptions can be complex to explain.This talk will explain these assumptions for data scientists and analysts without in-depth training of causal inference so they can use and explain these methods more confidently to change people's minds using data.Instead of relying solely on randomized control trials (also known as A/B tests), which are considered the gold standard for inferring causality, data scientists and analysts are increasingly turning to quasi-experimental methods to make recommendations based on causality.These methods, including open-source libraries such as CausalImpact (originally an R package but with numerous Python ports), are easy to use, but their assumptions can be complex to explain.\", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n') score=0.5365977414214874\n",
            "\n",
            "node=TextNode(id_='d3a09695-68b5-40c8-b3dc-6b6ab31367fb', embedding=None, metadata={'title': \"Causal Inference Libraries: What They Do, What I'd Like Them To Do\", 'speakers': 'Kevin Klein'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='0b21795f-fb6f-583d-a77d-bceee45fd768', node_type=None, metadata={'title': \"Causal Inference Libraries: What They Do, What I'd Like Them To Do\", 'speakers': 'Kevin Klein'}, hash='ba37eaf63d2629ea1fbf3345af76e1dc3a797adf175144f8e9fe10a87eecd8f9'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='f3794d8b-b455-411f-9dc2-745e533e43d9', node_type=None, metadata={'title': \"Causal Inference Libraries: What They Do, What I'd Like Them To Do\", 'speakers': 'Kevin Klein'}, hash='d9021be6a89c321a2507dc5b592fd65d21309941dd5df982d660b2faea511257')}, hash='cadcc8444800ed39cb34f89dafefaa2b06f9a173d45b07d8d05061aadfc240c4', text='They will know which libraries to use for different types of data and which methods are most appropriate for different scenarios.This talk could be particularly relevant for Data Scientists wishing to analyze experiments, such as A/B tests, or trying to derive causal statements from observational, non-experimental data.Participants are not expected to have Causal Inference expertise.Yet, a fundamental understanding of Machine Learning and Probability Theory will be beneficial.0-5\u2019: Why Causal Inference and why CATE estimation?5-10\u2019: What are some conceptual ways of estimating CATEs?10-20\u2019: How can we use EconML and CausalML for CATE estimation on a real dataset?20-30\u2019: What are we missing from EconML and CausalML?', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n') score=0.47913421453638927\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Retrieve chunks related to causal machine learning\n",
        "results = retriever.retrieve(\"causal\")\n",
        "for result in results:\n",
        "    print(result)\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc675732-2371-4a41-8a11-4c00577bd4e8",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'title': 'The proof of the pudding is in the (way of) eating: quasi-experimental methods of causal inference and their practical pitfalls', 'speakers': 'Jakob Willisch'}\n",
            "\n",
            "{'title': \"Causal Inference Libraries: What They Do, What I'd Like Them To Do\", 'speakers': 'Kevin Klein'}\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Print the talk title and speaker for results related to causal machine learning\n",
        "for result in results:\n",
        "    print(result.node.metadata)\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "11b98123-b560-4bc9-b611-3f2db8705262",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create a new retriever that retrieves more than 2 results. Hint: [llama-index Retriever documentation](https://gpt-index.readthedocs.io/en/v0.8.5.post2/api_reference/query/retrievers/vector_store.html)\n",
        "retriever = index.as_retriever(similarity_top_k=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "84f418a4-19d4-4ec0-b102-714de283c7fc",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{(\"Causal Inference Libraries: What They Do, What I'd Like Them To Do\",\n",
            "  'Kevin Klein'),\n",
            " ('Personalization at Uber scale via causal-driven machine learning',\n",
            "  'Okke van der Wal'),\n",
            " ('Staggered Difference-in-Differences in Practice: Causal Insights from the '\n",
            "  'Music Industry',\n",
            "  'Nazli M. Alagoz'),\n",
            " ('The proof of the pudding is in the (way of) eating: quasi-experimental '\n",
            "  'methods of causal inference and their practical pitfalls',\n",
            "  'Jakob Willisch')}\n"
          ]
        }
      ],
      "source": [
        "# Find all talks about causal inference at PyData\n",
        "results = retriever.retrieve('causal')\n",
        "talks = set([(r.node.metadata['title'], r.node.metadata['speakers']) for r in results])\n",
        "pprint.pprint(talks)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f81ee400-d0c5-4ad2-b68e-09c84ce4044c",
      "metadata": {},
      "source": [
        "## Querying the vector index with an external LLM"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "601ced37-8de0-4d5e-8abb-6fae18bf1803",
      "metadata": {},
      "source": [
        "### Set OpenAI API key (optional)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "62f5b922-be03-435f-b581-053dfb47e5d7",
      "metadata": {},
      "outputs": [],
      "source": [
        "from secret import openai_api_key"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "883fc70c-f18f-412c-bae5-3ac06a090a92",
      "metadata": {},
      "source": [
        "### Use OpenAI's gpt-3.5-turbo for querying"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d4d6ce8a-85d1-4d29-a182-a749782d14c3",
      "metadata": {},
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "name 'openai_api_key' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "Cell \u001b[0;32mIn[18], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m llm \u001b[38;5;241m=\u001b[39m llama_index\u001b[38;5;241m.\u001b[39mllms\u001b[38;5;241m.\u001b[39mOpenAI(model\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mgpt-3.5-turbo\u001b[39m\u001b[38;5;124m\"\u001b[39m, api_key\u001b[38;5;241m=\u001b[39m\u001b[43mopenai_api_key\u001b[49m)\n\u001b[1;32m      2\u001b[0m service_context \u001b[38;5;241m=\u001b[39m llama_index\u001b[38;5;241m.\u001b[39mServiceContext\u001b[38;5;241m.\u001b[39mfrom_defaults(\n\u001b[1;32m      3\u001b[0m   embed_model\u001b[38;5;241m=\u001b[39membed_model, chunk_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m256\u001b[39m, llm\u001b[38;5;241m=\u001b[39mllm\n\u001b[1;32m      4\u001b[0m )\n",
            "\u001b[0;31mNameError\u001b[0m: name 'openai_api_key' is not defined"
          ]
        }
      ],
      "source": [
        "llm = llama_index.llms.OpenAI(model=\"gpt-3.5-turbo\", api_key=openai_api_key)\n",
        "service_context = llama_index.ServiceContext.from_defaults(\n",
        "  embed_model=embed_model, chunk_size=256, llm=llm\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "54c91e61-7b94-4a1c-a68b-b22553e8b78d",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-08-30T16:38:04.830107+0200 - INFO - Loaded index from local storage\n"
          ]
        }
      ],
      "source": [
        "if not os.path.exists(INDEX_PATH):\n",
        "    logger.error(\"Index file for part 1 does not exist on disk. :(\")\n",
        "else:\n",
        "    try:                                                                             \n",
        "        storage_context = llama_index.StorageContext.from_defaults(persist_dir=INDEX_PATH)\n",
        "        index = llama_index.load_index_from_storage(storage_context, service_context=service_context)\n",
        "        logger.info(\"Loaded index from local storage\")                               \n",
        "    except Exception as e:                                                           \n",
        "        logger.error(e) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "de9d00b1-c133-4af2-b9f7-0e82113797f3",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create a query engine\n",
        "query_engine = index.as_query_engine()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "64136237-b60d-48b5-89d7-e749b71e061d",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Find talks that might be interesting for startup founders\n",
        "response = query_engine.query(\"Which talks are probably interesting for startup founders?\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c9ad3806-ad31-4774-a2f4-fbf085d48a9b",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Context information is below.\n",
            "---------------------\n",
            "title: Kickstart AI sponsored drinks [time & location TBD]\n",
            "\n",
            "Kickstart AI sponsored drinks [time & location TBD]\n",
            "\n",
            "Kickstart AI is a foundation powered by a coalition of iconic Dutch brands (Ahold Delhaize, ING, KLM and NS). Their mission is to accelerate AI adoption in the Netherlands, and improve society through the use of AI.\n",
            "\n",
            "Lorem ipsum dolor\n",
            "\n",
            "title: LLM Agents 101: How I Gave ChatGPT Access to My To-Do List\n",
            "\n",
            "We'll showcase these solutions through amusing moments and challenges encountered during the development of my to-do list agent.- A summary of what works well and what still needs improvement\n",
            "\n",
            "This talk is designed as an introduction to LLM agents.Throughout the presentation, I will aim to maintain a high-level perspective to ensure that even less technical audience members can grasp the concepts.To achieve this, I will share entertaining situations where my agent did something unexpected and how I resolved those issues.However, the presentation will still be practical enough for technical people to know all the concepts and techniques to develop their LLM agents.\n",
            "---------------------\n",
            "Given the context information and not prior knowledge, answer the query.\n",
            "Query: Which talks are probably interesting for startup founders?\n",
            "Answer: \n"
          ]
        }
      ],
      "source": [
        "print(response.response)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc130808-7ec1-4ec0-aeaf-92b2a4ec72ac",
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "id": "c913d2a2-f87f-4505-abb4-49e6b2291e52",
      "metadata": {},
      "source": [
        "# Part 2: Building a custom vector index"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "47bccc4d-52f3-4610-afae-1b08278dad37",
      "metadata": {},
      "source": [
        "How to use this notebook: \n",
        "1. Execute all cells under \"Setup\"\n",
        "2. Fill in the exercises below"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e03ae8f4-83b1-49a8-bf1e-19180366a102",
      "metadata": {},
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "038506f9-dcca-400c-b10b-5e7e612b1792",
      "metadata": {},
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "effd39ab-b6ba-4d69-8e8f-39f382d7712f",
      "metadata": {},
      "outputs": [],
      "source": [
        "import pprint\n",
        "import json\n",
        "import os\n",
        "import sys\n",
        "from pathlib import Path\n",
        "from loguru import logger\n",
        "import llama_index"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a4bd8c85-03d8-4894-8dfe-b419d4f6b6f2",
      "metadata": {},
      "source": [
        "### Logging"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5f44daa1-a3e6-45bf-bc6e-8c84fad646b2",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "logger.remove()\n",
        "logger.add(sys.stdout, format=\"{time} - {level} - {message}\", level=\"DEBUG\")\n",
        "logger.add(\"tutorial_part_1.log\", level=\"DEBUG\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "84e2e8b1-5b94-4dcf-b109-9e8351b087b4",
      "metadata": {},
      "outputs": [],
      "source": [
        "DATA_PATH = Path(\"data/julia_evans/blogposts.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "da94c113-deb9-49fc-a072-3f82f82afadd",
      "metadata": {},
      "source": [
        "### Load Julia Evans blogpost data from file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a5876b6c-0373-43d4-8f28-26b40dbf5f2a",
      "metadata": {},
      "outputs": [],
      "source": [
        "with open(DATA_PATH, 'r') as infile:\n",
        "    blogposts = json.loads(infile.read())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c02640e6-afcf-4eb6-a5e7-e89cf7914cd3",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-09-04T17:07:40.027379+0200 - INFO - Loaded 20 blogposts from file.\n"
          ]
        }
      ],
      "source": [
        "logger.info(f\"Loaded {len(blogposts)} blogposts from file.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "db2b0d29-ce87-454f-b7b8-e6ddd3a52c87",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1: Notes on using a single-person Mastodon server\n",
            "2: What helps people get comfortable on the command line?\n",
            "3: Some tactics for writing in public\n",
            "4: Behind \"Hello World\" on Linux\n",
            "5: Why is DNS still hard to learn?\n",
            "6: Lima: a nice way to run Linux VMs on Mac\n",
            "7: Open sourcing the nginx playground\n",
            "8: New zine: How Integers and Floats Work\n",
            "9: Some blogging myths\n",
            "10: New playground: memory spy\n",
            "11: Introducing \"Implement DNS in a Weekend\"\n",
            "12: New talk: Learning DNS in 10 years\n",
            "13: New playground: integer.exposed\n",
            "14: A list of programming playgrounds\n",
            "15: Building a custom site for zine feedback\n",
            "16: Some possible reasons for 8-bit bytes\n",
            "17: How do Nix builds work?\n",
            "18: Some notes on using nix\n",
            "19: Writing Javascript without a build system\n",
            "20: Print copies of The Pocket Guide to Debugging have arrived\n"
          ]
        }
      ],
      "source": [
        "for idx, post in enumerate(blogposts):\n",
        "    print(f\"{idx+1}: {post['title']}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "867fe918-339e-4de3-a614-f73a44f4b949",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Example blogpost:\n",
            "{'author': 'Julia Evans',\n",
            " 'guidislink': False,\n",
            " 'id': 'https://jvns.ca/blog/2023/08/11/some-notes-on-mastodon/',\n",
            " 'link': 'https://jvns.ca/blog/2023/08/11/some-notes-on-mastodon/',\n",
            " 'text': \"I started using Mastodon back in November, and it's the Twitter \"\n",
            "         \"alternative where I've been spending most of my time recently, \"\n",
            "         'mostly because the Fediverse is where a lot of the Linux nerds seem '\n",
            "         'to be right now.\\n'\n",
            "         '\\n'\n",
            "         \"I've found Mastodon quite a bit more confusing than Twitter because \"\n",
            "         \"it's a distributed system, so here are a few technical things I've \"\n",
            "         \"learned about it over the last 10 months. I'll mostly talk about \"\n",
            "         'what using a single-person server has been like for me, as well as a '\n",
            "         'couple of notes about the API, DMs and ActivityPub.\\n'\n",
            "         '\\n'\n",
            "         \"I might have made some mistakes, please let me know if I've gotten \"\n",
            "         'anything wrong!\\n'\n",
            "         '\\n'\n",
            "         \"### what's a mastodon instance?\\n\"\n",
            "         '\\n'\n",
            "         'First: Mastodon is a decentralized collection of independently run '\n",
            "         'servers instead of One Big Server. The software is [open '\n",
            "         'source](https://github.com/mastodon/mastodon).\\n'\n",
            "         '\\n'\n",
            "         'In general, if you have an account on one server (like '\n",
            "         '`ruby.social`), you **can** follow people on another server (like '\n",
            "         '`hachyderm.io`), and they can follow you.\\n'\n",
            "         '\\n'\n",
            "         'I\\'m going to use the terms \"Mastodon server\" and \"Mastodon '\n",
            "         'instance\" interchangeably in this post.\\n'\n",
            "         '\\n'\n",
            "         '### on choosing a Mastodon instance\\n'\n",
            "         '\\n'\n",
            "         'These were the things I was concerned about when choosing an '\n",
            "         'instance:\\n'\n",
            "         '\\n'\n",
            "         '  1. An instance name that I was comfortable being part of my online '\n",
            "         \"identity. For example, I probably wouldn't want to be \"\n",
            "         \"`@b0rk@infosec.exchange` because I'm not an infosec person.\\n\"\n",
            "         \"  2. The server's stability. Most servers are volunteer-run, and \"\n",
            "         'volunteer moderation work can be exhausting - will the server really '\n",
            "         'be around in a few years? For example '\n",
            "         '[mastodon.technology](https://ashfurrow.com/blog/mastodon-technology-shutdown/) '\n",
            "         'and mastodon.lol shut down.\\n'\n",
            "         \"  3. The admins' moderation policies.\\n\"\n",
            "         \"  4. That server's general reputation with other servers. I started \"\n",
            "         'out on `mastodon.social`, but some servers choose to block or limit '\n",
            "         'mastodon.social for various reasons\\n'\n",
            "         '  5. The community: every Mastodon instance has a local timeline '\n",
            "         'with all posts from users on that instance, would I be interested in '\n",
            "         'reading the local timeline?\\n'\n",
            "         '  6. Whether my account would be a burden for the admin of that '\n",
            "         'server (since I have a lot of followers)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'In the end, I chose to run my own mastodon server because it seemed '\n",
            "         \"simplest - I could pick a domain I liked, and I knew I'd definitely \"\n",
            "         \"agree with the moderation decisions because I'd be in charge.\\n\"\n",
            "         '\\n'\n",
            "         \"I'm not going to give server recommendations here, but here's a list \"\n",
            "         'of the [top 200 most common servers people who follow me '\n",
            "         'use](https://gist.github.com/jvns/5eb0a58319b93049a8c014433766edd3).\\n'\n",
            "         '\\n'\n",
            "         '### using your own domain\\n'\n",
            "         '\\n'\n",
            "         'One big thing I wondered was - can I use my own domain (and have the '\n",
            "         \"username `@b0rk@jvns.ca` or something) but be on someone else's \"\n",
            "         'Mastodon server?\\n'\n",
            "         '\\n'\n",
            "         'The answer to this seems to be basically \"no\": if you want to use '\n",
            "         'your own domain on Mastodon, you need to run your own server. (you '\n",
            "         'can [kind of do '\n",
            "         'this](https://blog.maartenballiauw.be/post/2022/11/05/mastodon-own-donain-without-hosting-server.html), '\n",
            "         \"but it's more like an alias or redirect - if I used that method to \"\n",
            "         'direct `b0rk@jvns.ca` to `b0rk@mastodon.social`, my posts would '\n",
            "         'still show up as being from `b0rk@mastodon.social`)\\n'\n",
            "         '\\n'\n",
            "         \"There's also other ActivityPub software \"\n",
            "         '([Takah\u0113](https://jointakahe.org/)) that supports people bringing '\n",
            "         'their own domain in a first-class way.\\n'\n",
            "         '\\n'\n",
            "         '### notes on having my own server\\n'\n",
            "         '\\n'\n",
            "         'I really wanted to have a way to use my own domain name for '\n",
            "         'identity, but to share server hosting costs with other people. This '\n",
            "         \"isn't possible on Mastodon right now, so I decided to set up my own \"\n",
            "         'server instead.\\n'\n",
            "         '\\n'\n",
            "         'I chose to run a Mastodon server (instead of some other ActivityPub '\n",
            "         'implementation) because Mastodon is the most popular one. Good '\n",
            "         'managed Mastodon hosting is readily available, there are tons of '\n",
            "         'options for client apps, and I know for sure that my server will '\n",
            "         \"work well with other people's servers.\\n\"\n",
            "         '\\n'\n",
            "         'I use [masto.host](https://masto.host/) for Mastodon hosting, and '\n",
            "         \"it's been great so far. I have nothing interesting to say about what \"\n",
            "         \"it's like to operate a Mastodon instance because I know literally \"\n",
            "         'nothing about it. Masto.host handles all of the server '\n",
            "         'administration and Mastodon updates, and I never think about it at '\n",
            "         'all.\\n'\n",
            "         '\\n'\n",
            "         'Right now I\\'m on their $19/month (\"Star\") plan, but it\\'s possible '\n",
            "         'I could use a smaller plan with no problems. Right now their '\n",
            "         'cheapest plan is $6/month and I expect that would be fine for '\n",
            "         'someone with a smaller account.\\n'\n",
            "         '\\n'\n",
            "         'Some things I was worried about when embarking on my own Mastodon '\n",
            "         'server:\\n'\n",
            "         '\\n'\n",
            "         '  * I wanted to run the server at `social.jvns.ca`, but I wanted my '\n",
            "         'username to be `b0rk@jvns.ca` instead of `b0rk@social.jvns.ca`. To '\n",
            "         'get this to work I followed these [Setting up a personal fediverse '\n",
            "         'ID](https://jacobian.org/til/my-mastodon-instance/) directions from '\n",
            "         \"Jacob Kaplan-Moss and it's been fine.\\n\"\n",
            "         '  * The administration burden of running my own server. I imported a '\n",
            "         \"small list of servers to block/defederate from but didn't do \"\n",
            "         \"anything else. That's been fine.\\n\"\n",
            "         \"  * Reply and profile visibility. This has been annoying and we'll \"\n",
            "         'talk about it next\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### downsides to being on a single-person server\\n'\n",
            "         '\\n'\n",
            "         'Being on a 1-person server has some significant downsides. To '\n",
            "         'understand why, you need to understand a little about how Mastodon '\n",
            "         'works.\\n'\n",
            "         '\\n'\n",
            "         'Every Mastodon server has a database of posts. Servers only have '\n",
            "         'posts that they were explicitly sent by another server in their '\n",
            "         'database.\\n'\n",
            "         '\\n'\n",
            "         'Some reasons that servers might receive posts:\\n'\n",
            "         '\\n'\n",
            "         '  * someone on the server follows a user\\n'\n",
            "         '  * a post mentions someone on the server\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'As a 1-person server, my server does not receive that many posts! I '\n",
            "         'only get posts from people I follow or posts that explicitly mention '\n",
            "         'me in some way.\\n'\n",
            "         '\\n'\n",
            "         'The causes several problems:\\n'\n",
            "         '\\n'\n",
            "         \"  1. when I visit someone's profile on Mastodon who I don't already \"\n",
            "         \"follow, my server will not fetch the profile's content (it'll fetch \"\n",
            "         'their profile picture, description, and pinned posts, but not any of '\n",
            "         \"their post history). So their profile appears as if they've never \"\n",
            "         'posted anything\\n'\n",
            "         '  2. bad reply visibility: when I look at the replies to somebody '\n",
            "         \"else's post (even if I follow them!), I don't see all of the \"\n",
            "         'replies, only the ones which have made it to my server. If you want '\n",
            "         'to understand the exact rules about who can see which replies (which '\n",
            "         \"are quite complicated!), [here's a great deep \"\n",
            "         'dive](https://seb.jambor.dev/posts/understanding-activitypub-part-3-the-state-of-mastodon/) '\n",
            "         \"by Sebastian Jambor. I think it's possible to end up in a state \"\n",
            "         'where no one person can see all of the replies, including the '\n",
            "         'original poster.\\n'\n",
            "         '  3. favourite and boost accounts are inaccurate - usually posts '\n",
            "         'show up having at most 1 or 2 favourites / boosts, even if the post '\n",
            "         'was actually favourite or boosted hundreds of times. I think this is '\n",
            "         'because it only counts favourites/boosts from people I follow.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'All of these things will happen to users of any small Mastodon '\n",
            "         'server, not just 1-person servers.\\n'\n",
            "         '\\n'\n",
            "         '### bad reply visibility makes conversations harder\\n'\n",
            "         '\\n'\n",
            "         \"A lot of people are on smaller servers, so when they're \"\n",
            "         \"participating in a conversation, they can't see all the replies to \"\n",
            "         'the post.\\n'\n",
            "         '\\n'\n",
            "         'This means that replies can get pretty repetitive because people '\n",
            "         \"literally cannot see each other's replies. This is especially \"\n",
            "         'annoying for posts that are popular or controversial, because the '\n",
            "         'person who made the post has to keep reading similar replies over '\n",
            "         \"and over again by people who think they're making the point for the \"\n",
            "         'first time.\\n'\n",
            "         '\\n'\n",
            "         'To get around this (as a reader), you can click \"open link to post\" '\n",
            "         'or something in your Mastodon client, which will open up the page on '\n",
            "         \"the poster's server where you can read all of the replies. It's \"\n",
            "         'pretty annoying though.\\n'\n",
            "         '\\n'\n",
            "         \"As a poster, I've tried to reduce repetitiveness in replies by:\\n\"\n",
            "         '\\n'\n",
            "         '  * putting requests in my posts like \"(no need to reply if you '\n",
            "         'don\u2019t remember, or if you\u2019ve been using the command line comfortably '\n",
            "         'for 15 years \u2014 this question isn\u2019t for you :) )\"\\n'\n",
            "         '  * occasionally editing my posts to include very common replies\\n'\n",
            "         '  * very occasionally deleting the post if it gets too out of hand\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'The Mastodon devs are extremely aware of these issues, there are a '\n",
            "         'bunch of github issues about them:\\n'\n",
            "         '\\n'\n",
            "         '  * [backfill statuses when first '\n",
            "         'subscribed](https://github.com/mastodon/mastodon/issues/34)\\n'\n",
            "         '  * [fetch whole conversation '\n",
            "         'threads](https://github.com/mastodon/mastodon/issues/9409)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'My guess is that there are technical reasons these features are '\n",
            "         'difficult to add because those issues have been open for 5-7 years.\\n'\n",
            "         '\\n'\n",
            "         'The Mastodon devs have said that they plan to improve reply '\n",
            "         'fetching, but that it requires a significant amount of work.\\n'\n",
            "         '\\n'\n",
            "         '### some visibility workarounds\\n'\n",
            "         '\\n'\n",
            "         'Some people have built workarounds for fetching profiles / replies.\\n'\n",
            "         '\\n'\n",
            "         '  * [Fedifetcher](https://blog.thms.uk/fedifetcher)\\n'\n",
            "         '  * [combine.social](https://combine.social/)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'Also, there are a couple of Mastodon clients which will proactively '\n",
            "         'fetch replies. For iOS:\\n'\n",
            "         '\\n'\n",
            "         '  * Mammoth does it automatically\\n'\n",
            "         '  * Mona will fetch posts if I click \"load from remote server\" '\n",
            "         'manually\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I haven't tried those yet though.\\n\"\n",
            "         '\\n'\n",
            "         '### other downsides of running your own server: discovery is much '\n",
            "         'harder\\n'\n",
            "         '\\n'\n",
            "         'Mastodon instances have a \"local timeline\" where you can see '\n",
            "         'everything other people on the server are posting, and a \"federated '\n",
            "         'timeline\" which shows sort of a combined feed from everyone followed '\n",
            "         'by anyone on the server. This means that you can see trending posts '\n",
            "         \"and get an idea of what's going on and find people to follow. You \"\n",
            "         \"don't get that if you're on a 1-person server - it's just me talking \"\n",
            "         'to myself! (plus occasional interjections from [my reruns '\n",
            "         'bot](https://social.jvns.ca/@b0rk_reruns)).\\n'\n",
            "         '\\n'\n",
            "         'Some workarounds people mentioned for this:\\n'\n",
            "         '\\n'\n",
            "         '  * you can populate your federated timeline with posts from another '\n",
            "         \"instance by using a **relay**. I haven't done this but someone else \"\n",
            "         'said they use [FediBuzz](https://relay.fedi.buzz) and I might try it '\n",
            "         'out.\\n'\n",
            "         '  * some mastodon clients (like apparently Moshidon on Android) let '\n",
            "         'you follow other instances\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'If anyone else on small servers has suggestions for how to make '\n",
            "         \"discovery easier I'd love to hear them.\\n\"\n",
            "         '\\n'\n",
            "         '### account migration\\n'\n",
            "         '\\n'\n",
            "         'When I moved to my own server from `mastodon.social`, I needed to '\n",
            "         \"run an account migration to move over my followers. First, here's \"\n",
            "         'how migration works:\\n'\n",
            "         '\\n'\n",
            "         '  1. Account migration **does not** move over your posts. All of my '\n",
            "         'posts stayed on my old account. This is part of why I moved to '\n",
            "         \"running my own server - I didn't want to ever lose my posts a second \"\n",
            "         'time.\\n'\n",
            "         '  2. Account migration **does not** move over the list of people you '\n",
            "         'follow/mute/block. But you can import/export that list in your '\n",
            "         \"Mastodon settings so it's not a big deal. If you follow private \"\n",
            "         \"accounts they'll have to re-approve your follow request.\\n\"\n",
            "         '  3. Account migration **does** move over your followers\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"The follower move was the part I was most worried about. Here's how \"\n",
            "         'it turned out:\\n'\n",
            "         '\\n'\n",
            "         '  * over ~24 hours, most of my followers moved to the new account\\n'\n",
            "         '  * one or two servers did not get the message about the account '\n",
            "         'migration for some reason, so about 2000 followers were \"stuck\" and '\n",
            "         \"didn't migrate. I fixed this by waiting 30 days and re-running the \"\n",
            "         'account migration, which moved over most of the remaining followers. '\n",
            "         \"There's also a [tootctl \"\n",
            "         'command](https://github.com/mastodon/mastodon/issues/22281) that the '\n",
            "         'admin of the **old instance** can run to retry the migration\\n'\n",
            "         '  * about 200 of my followers never migrated over, I think because '\n",
            "         \"they're using ActivityPub software other than Mastodon which doesn't \"\n",
            "         'support account migration. You can see the [old account '\n",
            "         'here](https://mastodon.social/@b0rk)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### using the Mastodon API is great\\n'\n",
            "         '\\n'\n",
            "         \"One thing I love about Mastodon is - it has an API that's MUCH \"\n",
            "         \"easier to use than Twitter's API. I've always been frustrated with \"\n",
            "         'how difficult it is to navigate large Twitter threads, so I made a '\n",
            "         'small [mastodon thread view](https://mastodon-thread-view.jvns.ca/) '\n",
            "         \"website that lets you log into your Mastodon account. It's pretty \"\n",
            "         \"janky and it's really only made for me to use, but I've really \"\n",
            "         'appreciated the ability to write my own janky software to improve my '\n",
            "         'Mastodon experience.\\n'\n",
            "         '\\n'\n",
            "         'Some notes on the Mastodon API:\\n'\n",
            "         '\\n'\n",
            "         '  * You can build Mastodon client software totally on the frontend '\n",
            "         'in Javascript, which is really cool.\\n'\n",
            "         \"  * I couldn't find a vanilla Javascript Mastodon client, so I \"\n",
            "         '[wrote a crappy '\n",
            "         'one](https://github.com/jvns/mastodon-threaded-replies/blob/main/mastodon.js)\\n'\n",
            "         '  * [API docs are '\n",
            "         'here](https://docs.joinmastodon.org/client/intro/)\\n'\n",
            "         \"  * Here's a [tiny Python script I used to list all my Mastodon \"\n",
            "         'followers](https://gist.github.com/jvns/0fe51383cbbb63e94177c60f1e0371c6), '\n",
            "         'which also serves as a simple example of how easy using the API is.\\n'\n",
            "         '  * The best documentation I could find for which OAuth scopes '\n",
            "         'correspond to which API endpoints is [this github '\n",
            "         'issue](https://github.com/mastodon/mastodon/pull/7929)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"Next I'll talk about a few general things about Mastodon that \"\n",
            "         \"confused or surprised me that aren't specific to being on a \"\n",
            "         'single-person instance.\\n'\n",
            "         '\\n'\n",
            "         '### DMs are weird\\n'\n",
            "         '\\n'\n",
            "         'The way Mastodon DMs work surprised me in a few ways:\\n'\n",
            "         '\\n'\n",
            "         '  * Technically DMs are just regular posts with visibility limited '\n",
            "         'to the people mentioned in the post. This means that if you '\n",
            "         'accidentally mention someone in a DM (\"@x is such a jerk\"), it\\'s '\n",
            "         'possible to accidentally send the message to them\\n'\n",
            "         \"  * DMs aren't very private: the admins on the sending and receiving \"\n",
            "         'servers can technically read your DMs if they have access to the '\n",
            "         \"database. So they're not appropriate for sensitive information.\\n\"\n",
            "         \"  * Turning off DMs is weird. Personally I don't like receiving DMs \"\n",
            "         \"from strangers - it's too much to keep track of and I'd prefer that \"\n",
            "         \"people email me. On Twitter, I can just turn it off and people won't \"\n",
            "         'see an option to DM me. But on Mastodon, when I turn off '\n",
            "         'notifications for DMs, anyone can still \"DM\" me, but the message '\n",
            "         \"will go into a black hole and I'll never see it. I put a note in my \"\n",
            "         'profile about this.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### defederation and limiting\\n'\n",
            "         '\\n'\n",
            "         'There are a couple of different ways for a server to block another '\n",
            "         \"Mastodon server. I haven't really had to do this much but people \"\n",
            "         'talk about it a lot and I was confused about the difference, so:\\n'\n",
            "         '\\n'\n",
            "         '  * A server can **defederate** from another server (this seems to '\n",
            "         'be called '\n",
            "         '[suspend](https://docs.joinmastodon.org/admin/moderation/#suspend-server) '\n",
            "         'in the Mastodon docs). This means that nobody on a server can follow '\n",
            "         'someone from the other server.\\n'\n",
            "         '  * A server can '\n",
            "         '**[limit](https://docs.joinmastodon.org/admin/moderation/#limit-user)** '\n",
            "         '(also known as \"silence\") a user or server. This means that content '\n",
            "         \"from that user is only visible to that user's followers - people \"\n",
            "         'can\\'t discover the user through retweets (aka \"boosts\" on '\n",
            "         'Mastodon).\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"One thing that wasn't obvious to me is that who servers defederate / \"\n",
            "         \"limit is sometimes hidden, so it's hard to suss out what's going on \"\n",
            "         \"if you're considering joining a server, or trying to understand why \"\n",
            "         \"you can't see certain posts.\\n\"\n",
            "         '\\n'\n",
            "         \"### there's no search for posts\\n\"\n",
            "         '\\n'\n",
            "         \"There's no way to search past posts you've read. If I see something \"\n",
            "         'interesting on my timeline and want to find it later, I usually '\n",
            "         \"can't. (Mastodon has a [Elasticsearch-based search \"\n",
            "         'feature](https://docs.joinmastodon.org/admin/optional/elasticsearch/), '\n",
            "         'but it only allows you to search your own posts, your mentions, your '\n",
            "         'favourites, and your bookmarks)\\n'\n",
            "         '\\n'\n",
            "         'These limitations on search are intentional (and a very common '\n",
            "         \"source of arguments) - it's a privacy / safety issue. Here's a \"\n",
            "         '[summary from Tim '\n",
            "         'Bray](https://www.tbray.org/ongoing/When/202x/2022/12/30/Mastodon-Privacy-and-Search) '\n",
            "         'with lots of links.\\n'\n",
            "         '\\n'\n",
            "         'It would be personally convenient for me to be able to search more '\n",
            "         \"easily but I respect folks' safety concerns so I'll leave it at \"\n",
            "         'that.\\n'\n",
            "         '\\n'\n",
            "         'My understanding is that the Mastodon devs are planning to add '\n",
            "         'opt-in search for public posts relatively soon.\\n'\n",
            "         '\\n'\n",
            "         '### other ActivityPub software\\n'\n",
            "         '\\n'\n",
            "         \"We've been talking about Mastodon a lot, but not everyone who I \"\n",
            "         'follow is using Mastodon: Mastodon uses a protocol called '\n",
            "         '[ActivityPub](https://activitypub.rocks/) to distribute messages.\\n'\n",
            "         '\\n'\n",
            "         'Here are some examples of other software I see people talking about, '\n",
            "         'in no particular order:\\n'\n",
            "         '\\n'\n",
            "         '  * [Calckey](https://calckey.org/)\\n'\n",
            "         '  * [Akkoma](https://akkoma.social/)\\n'\n",
            "         '  * [gotosocial](https://gotosocial.org/)\\n'\n",
            "         '  * [Takah\u0113](https://jointakahe.org/)\\n'\n",
            "         '  * [writefreely](https://writefreely.org/)\\n'\n",
            "         '  * [pixelfed](https://pixelfed.org/) (for images)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I'm probably missing a bunch of important ones.\\n\"\n",
            "         '\\n'\n",
            "         \"### what's the difference between Mastodon and other ActivityPub \"\n",
            "         'software?\\n'\n",
            "         '\\n'\n",
            "         \"This confused me for a while, and I'm still not super clear on how \"\n",
            "         \"ActivityPub works. What I've understood is:\\n\"\n",
            "         '\\n'\n",
            "         '  * ActivityPub is a protocol (you can explore how it works with '\n",
            "         \"blinry's nice [JSON \"\n",
            "         'explorer](https://json.blinry.org/#https://chaos.social/users/blinry))\\n'\n",
            "         '  * Mastodon **servers** communicate with each other (and with other '\n",
            "         'ActivityPub servers) using ActivityPub\\n'\n",
            "         '  * Mastodon **clients** communicate with their server using the '\n",
            "         'Mastodon API, which is its own thing\\n'\n",
            "         \"  * There's also software like \"\n",
            "         '[GoToSocial](https://github.com/superseriousbusiness/gotosocial) '\n",
            "         'that aims to be compatible with the Mastodon API, so that you can '\n",
            "         'use a Mastodon client with it\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### more mastodon resources\\n'\n",
            "         '\\n'\n",
            "         '  * [Fedi.Tips](https://fedi.tips/) seems to be a great '\n",
            "         'introduction\\n'\n",
            "         '  * I think you can still use '\n",
            "         '[FediFinder](https://fedifinder.glitch.me/) to find folks you '\n",
            "         'followed on Twitter on Mastodon\\n'\n",
            "         \"  * I've been using the [Ivory](https://tapbots.com/ivory/) client \"\n",
            "         'on iOS, but there are lots of great clients. '\n",
            "         '[Elk](https://elk.zone/) is an alternative web client that folks '\n",
            "         'seem to like.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I haven't written here about what Mastodon culture is like because \"\n",
            "         'other people have done a much better job of talking about it than '\n",
            "         \"me, but of course it's is the biggest thing that affects your \"\n",
            "         'experience and it was the thing that took me longest to get a handle '\n",
            "         'on. A few links:\\n'\n",
            "         '\\n'\n",
            "         '  * Erin Kissane on [frictions people run into when joining '\n",
            "         'Mastodon](https://erinkissane.com/mastodon-is-easy-and-fun-except-when-it-isnt)\\n'\n",
            "         '  * Kyle Kingsbury wrote some great [moderation guidelines for '\n",
            "         'woof.group](https://blog.woof.group/mods/) (note: woof.group is a '\n",
            "         'LGBTQ+ leather instance, be prepared to see lots of NSFW posts if '\n",
            "         'you visit it)\\n'\n",
            "         '  * Mekka Okereke writes [lots of great posts about issues Black '\n",
            "         'people encounter on '\n",
            "         'Mastodon](https://hachyderm.io/@mekkaokereke/110273797004251326) '\n",
            "         \"(though they're all on Mastodon so it's a little hard to navigate)\\n\"\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"### that's all!\\n\"\n",
            "         '\\n'\n",
            "         \"I don't regret setting up a single-user server - even though it's \"\n",
            "         \"inconvenient, it's important to me to have control over my social \"\n",
            "         'media. I think \"have control over my social media\" is more important '\n",
            "         'to me than it is to most other people though, because I use '\n",
            "         'Twitter/Mastodon a lot for work.\\n'\n",
            "         '\\n'\n",
            "         \"I am happy that I didn't _start out_ on a single-user server though \"\n",
            "         '- I think it would have made getting started on Mastodon a lot more '\n",
            "         'difficult.\\n'\n",
            "         '\\n'\n",
            "         \"Mastodon is pretty rough around the edges sometimes but I'm able to \"\n",
            "         'have more interesting conversations about computers there than I am '\n",
            "         \"on Twitter (or Bluesky), so that's where I'm staying for now.\\n\",\n",
            " 'title': 'Notes on using a single-person Mastodon server',\n",
            " 'updated': '2023-08-11T08:13:16+00:00'}\n"
          ]
        }
      ],
      "source": [
        "print(\"Example blogpost:\")\n",
        "pprint.pprint(blogposts[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6b4f10b1-759c-459b-bece-1495c099cf62",
      "metadata": {},
      "source": [
        "### Create a service context\n",
        "- No OpenAI API calls\n",
        "- No large local LLM\n",
        "- Just use the smallest sentence-transformers embeddings model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c0bfd191-3038-4ad7-a073-58b93e94f8d1",
      "metadata": {},
      "source": [
        "- all-minilm-l6-v2 has a maximum size of 256 tokens\n",
        "- source: https://www.sbert.net/docs/pretrained_models.html#model-overview"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85531cae-9ec3-47f3-b4c1-e9df4cea39f4",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "LLM is explicitly disabled. Using MockLLM.\n"
          ]
        }
      ],
      "source": [
        "service_context = llama_index.ServiceContext.from_defaults(\n",
        "  embed_model=\"local:sentence-transformers/all-minilm-l6-v2\", chunk_size=256, llm=None\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0fb5492d-e290-461a-a3f4-55f23818fee1",
      "metadata": {},
      "source": [
        "### Create documents from the blogposts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a03e2bf3-5d46-40d1-9ba5-c6d8dda9d0c0",
      "metadata": {},
      "outputs": [],
      "source": [
        "documents = [llama_index.Document(text=blogpost['text']) for blogpost in blogposts]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e829de61-44ef-4701-b01d-e24bb34b6d78",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "20"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(documents)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "91772358-47dc-4e6a-861a-aa9e295805ca",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Example document:\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "Document(id_='2f48c07d-395c-46a7-8ceb-2e3eda7e8a99', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='74bb2056380d3e3871dfa6ae4f2b4d9af8e908962b533e58819a94d497caca5b', text='I started using Mastodon back in November, and it\\'s the Twitter alternative where I\\'ve been spending most of my time recently, mostly because the Fediverse is where a lot of the Linux nerds seem to be right now.\\n\\nI\\'ve found Mastodon quite a bit more confusing than Twitter because it\\'s a distributed system, so here are a few technical things I\\'ve learned about it over the last 10 months. I\\'ll mostly talk about what using a single-person server has been like for me, as well as a couple of notes about the API, DMs and ActivityPub.\\n\\nI might have made some mistakes, please let me know if I\\'ve gotten anything wrong!\\n\\n### what\\'s a mastodon instance?\\n\\nFirst: Mastodon is a decentralized collection of independently run servers instead of One Big Server. The software is [open source](https://github.com/mastodon/mastodon).\\n\\nIn general, if you have an account on one server (like `ruby.social`), you **can** follow people on another server (like `hachyderm.io`), and they can follow you.\\n\\nI\\'m going to use the terms \"Mastodon server\" and \"Mastodon instance\" interchangeably in this post.\\n\\n### on choosing a Mastodon instance\\n\\nThese were the things I was concerned about when choosing an instance:\\n\\n  1. An instance name that I was comfortable being part of my online identity. For example, I probably wouldn\\'t want to be `@b0rk@infosec.exchange` because I\\'m not an infosec person.\\n  2. The server\\'s stability. Most servers are volunteer-run, and volunteer moderation work can be exhausting - will the server really be around in a few years? For example [mastodon.technology](https://ashfurrow.com/blog/mastodon-technology-shutdown/) and mastodon.lol shut down.\\n  3. The admins\\' moderation policies.\\n  4. That server\\'s general reputation with other servers. I started out on `mastodon.social`, but some servers choose to block or limit mastodon.social for various reasons\\n  5. The community: every Mastodon instance has a local timeline with all posts from users on that instance, would I be interested in reading the local timeline?\\n  6. Whether my account would be a burden for the admin of that server (since I have a lot of followers)\\n\\n\\n\\nIn the end, I chose to run my own mastodon server because it seemed simplest - I could pick a domain I liked, and I knew I\\'d definitely agree with the moderation decisions because I\\'d be in charge.\\n\\nI\\'m not going to give server recommendations here, but here\\'s a list of the [top 200 most common servers people who follow me use](https://gist.github.com/jvns/5eb0a58319b93049a8c014433766edd3).\\n\\n### using your own domain\\n\\nOne big thing I wondered was - can I use my own domain (and have the username `@b0rk@jvns.ca` or something) but be on someone else\\'s Mastodon server?\\n\\nThe answer to this seems to be basically \"no\": if you want to use your own domain on Mastodon, you need to run your own server. (you can [kind of do this](https://blog.maartenballiauw.be/post/2022/11/05/mastodon-own-donain-without-hosting-server.html), but it\\'s more like an alias or redirect - if I used that method to direct `b0rk@jvns.ca` to `b0rk@mastodon.social`, my posts would still show up as being from `b0rk@mastodon.social`)\\n\\nThere\\'s also other ActivityPub software ([Takah\u0113](https://jointakahe.org/)) that supports people bringing their own domain in a first-class way.\\n\\n### notes on having my own server\\n\\nI really wanted to have a way to use my own domain name for identity, but to share server hosting costs with other people. This isn\\'t possible on Mastodon right now, so I decided to set up my own server instead.\\n\\nI chose to run a Mastodon server (instead of some other ActivityPub implementation) because Mastodon is the most popular one. Good managed Mastodon hosting is readily available, there are tons of options for client apps, and I know for sure that my server will work well with other people\\'s servers.\\n\\nI use [masto.host](https://masto.host/) for Mastodon hosting, and it\\'s been great so far. I have nothing interesting to say about what it\\'s like to operate a Mastodon instance because I know literally nothing about it. Masto.host handles all of the server administration and Mastodon updates, and I never think about it at all.\\n\\nRight now I\\'m on their $19/month (\"Star\") plan, but it\\'s possible I could use a smaller plan with no problems. Right now their cheapest plan is $6/month and I expect that would be fine for someone with a smaller account.\\n\\nSome things I was worried about when embarking on my own Mastodon server:\\n\\n  * I wanted to run the server at `social.jvns.ca`, but I wanted my username to be `b0rk@jvns.ca` instead of `b0rk@social.jvns.ca`. To get this to work I followed these [Setting up a personal fediverse ID](https://jacobian.org/til/my-mastodon-instance/) directions from Jacob Kaplan-Moss and it\\'s been fine.\\n  * The administration burden of running my own server. I imported a small list of servers to block/defederate from but didn\\'t do anything else. That\\'s been fine.\\n  * Reply and profile visibility. This has been annoying and we\\'ll talk about it next\\n\\n\\n\\n### downsides to being on a single-person server\\n\\nBeing on a 1-person server has some significant downsides. To understand why, you need to understand a little about how Mastodon works.\\n\\nEvery Mastodon server has a database of posts. Servers only have posts that they were explicitly sent by another server in their database.\\n\\nSome reasons that servers might receive posts:\\n\\n  * someone on the server follows a user\\n  * a post mentions someone on the server\\n\\n\\n\\nAs a 1-person server, my server does not receive that many posts! I only get posts from people I follow or posts that explicitly mention me in some way.\\n\\nThe causes several problems:\\n\\n  1. when I visit someone\\'s profile on Mastodon who I don\\'t already follow, my server will not fetch the profile\\'s content (it\\'ll fetch their profile picture, description, and pinned posts, but not any of their post history). So their profile appears as if they\\'ve never posted anything\\n  2. bad reply visibility: when I look at the replies to somebody else\\'s post (even if I follow them!), I don\\'t see all of the replies, only the ones which have made it to my server. If you want to understand the exact rules about who can see which replies (which are quite complicated!), [here\\'s a great deep dive](https://seb.jambor.dev/posts/understanding-activitypub-part-3-the-state-of-mastodon/) by Sebastian Jambor. I think it\\'s possible to end up in a state where no one person can see all of the replies, including the original poster.\\n  3. favourite and boost accounts are inaccurate - usually posts show up having at most 1 or 2 favourites / boosts, even if the post was actually favourite or boosted hundreds of times. I think this is because it only counts favourites/boosts from people I follow.\\n\\n\\n\\nAll of these things will happen to users of any small Mastodon server, not just 1-person servers.\\n\\n### bad reply visibility makes conversations harder\\n\\nA lot of people are on smaller servers, so when they\\'re participating in a conversation, they can\\'t see all the replies to the post.\\n\\nThis means that replies can get pretty repetitive because people literally cannot see each other\\'s replies. This is especially annoying for posts that are popular or controversial, because the person who made the post has to keep reading similar replies over and over again by people who think they\\'re making the point for the first time.\\n\\nTo get around this (as a reader), you can click \"open link to post\" or something in your Mastodon client, which will open up the page on the poster\\'s server where you can read all of the replies. It\\'s pretty annoying though.\\n\\nAs a poster, I\\'ve tried to reduce repetitiveness in replies by:\\n\\n  * putting requests in my posts like \"(no need to reply if you don\u2019t remember, or if you\u2019ve been using the command line comfortably for 15 years \u2014 this question isn\u2019t for you :) )\"\\n  * occasionally editing my posts to include very common replies\\n  * very occasionally deleting the post if it gets too out of hand\\n\\n\\n\\nThe Mastodon devs are extremely aware of these issues, there are a bunch of github issues about them:\\n\\n  * [backfill statuses when first subscribed](https://github.com/mastodon/mastodon/issues/34)\\n  * [fetch whole conversation threads](https://github.com/mastodon/mastodon/issues/9409)\\n\\n\\n\\nMy guess is that there are technical reasons these features are difficult to add because those issues have been open for 5-7 years.\\n\\nThe Mastodon devs have said that they plan to improve reply fetching, but that it requires a significant amount of work.\\n\\n### some visibility workarounds\\n\\nSome people have built workarounds for fetching profiles / replies.\\n\\n  * [Fedifetcher](https://blog.thms.uk/fedifetcher)\\n  * [combine.social](https://combine.social/)\\n\\n\\n\\nAlso, there are a couple of Mastodon clients which will proactively fetch replies. For iOS:\\n\\n  * Mammoth does it automatically\\n  * Mona will fetch posts if I click \"load from remote server\" manually\\n\\n\\n\\nI haven\\'t tried those yet though.\\n\\n### other downsides of running your own server: discovery is much harder\\n\\nMastodon instances have a \"local timeline\" where you can see everything other people on the server are posting, and a \"federated timeline\" which shows sort of a combined feed from everyone followed by anyone on the server. This means that you can see trending posts and get an idea of what\\'s going on and find people to follow. You don\\'t get that if you\\'re on a 1-person server - it\\'s just me talking to myself! (plus occasional interjections from [my reruns bot](https://social.jvns.ca/@b0rk_reruns)).\\n\\nSome workarounds people mentioned for this:\\n\\n  * you can populate your federated timeline with posts from another instance by using a **relay**. I haven\\'t done this but someone else said they use [FediBuzz](https://relay.fedi.buzz) and I might try it out.\\n  * some mastodon clients (like apparently Moshidon on Android) let you follow other instances\\n\\n\\n\\nIf anyone else on small servers has suggestions for how to make discovery easier I\\'d love to hear them.\\n\\n### account migration\\n\\nWhen I moved to my own server from `mastodon.social`, I needed to run an account migration to move over my followers. First, here\\'s how migration works:\\n\\n  1. Account migration **does not** move over your posts. All of my posts stayed on my old account. This is part of why I moved to running my own server - I didn\\'t want to ever lose my posts a second time.\\n  2. Account migration **does not** move over the list of people you follow/mute/block. But you can import/export that list in your Mastodon settings so it\\'s not a big deal. If you follow private accounts they\\'ll have to re-approve your follow request.\\n  3. Account migration **does** move over your followers\\n\\n\\n\\nThe follower move was the part I was most worried about. Here\\'s how it turned out:\\n\\n  * over ~24 hours, most of my followers moved to the new account\\n  * one or two servers did not get the message about the account migration for some reason, so about 2000 followers were \"stuck\" and didn\\'t migrate. I fixed this by waiting 30 days and re-running the account migration, which moved over most of the remaining followers. There\\'s also a [tootctl command](https://github.com/mastodon/mastodon/issues/22281) that the admin of the **old instance** can run to retry the migration\\n  * about 200 of my followers never migrated over, I think because they\\'re using ActivityPub software other than Mastodon which doesn\\'t support account migration. You can see the [old account here](https://mastodon.social/@b0rk)\\n\\n\\n\\n### using the Mastodon API is great\\n\\nOne thing I love about Mastodon is - it has an API that\\'s MUCH easier to use than Twitter\\'s API. I\\'ve always been frustrated with how difficult it is to navigate large Twitter threads, so I made a small [mastodon thread view](https://mastodon-thread-view.jvns.ca/) website that lets you log into your Mastodon account. It\\'s pretty janky and it\\'s really only made for me to use, but I\\'ve really appreciated the ability to write my own janky software to improve my Mastodon experience.\\n\\nSome notes on the Mastodon API:\\n\\n  * You can build Mastodon client software totally on the frontend in Javascript, which is really cool.\\n  * I couldn\\'t find a vanilla Javascript Mastodon client, so I [wrote a crappy one](https://github.com/jvns/mastodon-threaded-replies/blob/main/mastodon.js)\\n  * [API docs are here](https://docs.joinmastodon.org/client/intro/)\\n  * Here\\'s a [tiny Python script I used to list all my Mastodon followers](https://gist.github.com/jvns/0fe51383cbbb63e94177c60f1e0371c6), which also serves as a simple example of how easy using the API is.\\n  * The best documentation I could find for which OAuth scopes correspond to which API endpoints is [this github issue](https://github.com/mastodon/mastodon/pull/7929)\\n\\n\\n\\nNext I\\'ll talk about a few general things about Mastodon that confused or surprised me that aren\\'t specific to being on a single-person instance.\\n\\n### DMs are weird\\n\\nThe way Mastodon DMs work surprised me in a few ways:\\n\\n  * Technically DMs are just regular posts with visibility limited to the people mentioned in the post. This means that if you accidentally mention someone in a DM (\"@x is such a jerk\"), it\\'s possible to accidentally send the message to them\\n  * DMs aren\\'t very private: the admins on the sending and receiving servers can technically read your DMs if they have access to the database. So they\\'re not appropriate for sensitive information.\\n  * Turning off DMs is weird. Personally I don\\'t like receiving DMs from strangers - it\\'s too much to keep track of and I\\'d prefer that people email me. On Twitter, I can just turn it off and people won\\'t see an option to DM me. But on Mastodon, when I turn off notifications for DMs, anyone can still \"DM\" me, but the message will go into a black hole and I\\'ll never see it. I put a note in my profile about this.\\n\\n\\n\\n### defederation and limiting\\n\\nThere are a couple of different ways for a server to block another Mastodon server. I haven\\'t really had to do this much but people talk about it a lot and I was confused about the difference, so:\\n\\n  * A server can **defederate** from another server (this seems to be called [suspend](https://docs.joinmastodon.org/admin/moderation/#suspend-server) in the Mastodon docs). This means that nobody on a server can follow someone from the other server.\\n  * A server can **[limit](https://docs.joinmastodon.org/admin/moderation/#limit-user)** (also known as \"silence\") a user or server. This means that content from that user is only visible to that user\\'s followers - people can\\'t discover the user through retweets (aka \"boosts\" on Mastodon).\\n\\n\\n\\nOne thing that wasn\\'t obvious to me is that who servers defederate / limit is sometimes hidden, so it\\'s hard to suss out what\\'s going on if you\\'re considering joining a server, or trying to understand why you can\\'t see certain posts.\\n\\n### there\\'s no search for posts\\n\\nThere\\'s no way to search past posts you\\'ve read. If I see something interesting on my timeline and want to find it later, I usually can\\'t. (Mastodon has a [Elasticsearch-based search feature](https://docs.joinmastodon.org/admin/optional/elasticsearch/), but it only allows you to search your own posts, your mentions, your favourites, and your bookmarks)\\n\\nThese limitations on search are intentional (and a very common source of arguments) - it\\'s a privacy / safety issue. Here\\'s a [summary from Tim Bray](https://www.tbray.org/ongoing/When/202x/2022/12/30/Mastodon-Privacy-and-Search) with lots of links.\\n\\nIt would be personally convenient for me to be able to search more easily but I respect folks\\' safety concerns so I\\'ll leave it at that.\\n\\nMy understanding is that the Mastodon devs are planning to add opt-in search for public posts relatively soon.\\n\\n### other ActivityPub software\\n\\nWe\\'ve been talking about Mastodon a lot, but not everyone who I follow is using Mastodon: Mastodon uses a protocol called [ActivityPub](https://activitypub.rocks/) to distribute messages.\\n\\nHere are some examples of other software I see people talking about, in no particular order:\\n\\n  * [Calckey](https://calckey.org/)\\n  * [Akkoma](https://akkoma.social/)\\n  * [gotosocial](https://gotosocial.org/)\\n  * [Takah\u0113](https://jointakahe.org/)\\n  * [writefreely](https://writefreely.org/)\\n  * [pixelfed](https://pixelfed.org/) (for images)\\n\\n\\n\\nI\\'m probably missing a bunch of important ones.\\n\\n### what\\'s the difference between Mastodon and other ActivityPub software?\\n\\nThis confused me for a while, and I\\'m still not super clear on how ActivityPub works. What I\\'ve understood is:\\n\\n  * ActivityPub is a protocol (you can explore how it works with blinry\\'s nice [JSON explorer](https://json.blinry.org/#https://chaos.social/users/blinry))\\n  * Mastodon **servers** communicate with each other (and with other ActivityPub servers) using ActivityPub\\n  * Mastodon **clients** communicate with their server using the Mastodon API, which is its own thing\\n  * There\\'s also software like [GoToSocial](https://github.com/superseriousbusiness/gotosocial) that aims to be compatible with the Mastodon API, so that you can use a Mastodon client with it\\n\\n\\n\\n### more mastodon resources\\n\\n  * [Fedi.Tips](https://fedi.tips/) seems to be a great introduction\\n  * I think you can still use [FediFinder](https://fedifinder.glitch.me/) to find folks you followed on Twitter on Mastodon\\n  * I\\'ve been using the [Ivory](https://tapbots.com/ivory/) client on iOS, but there are lots of great clients. [Elk](https://elk.zone/) is an alternative web client that folks seem to like.\\n\\n\\n\\nI haven\\'t written here about what Mastodon culture is like because other people have done a much better job of talking about it than me, but of course it\\'s is the biggest thing that affects your experience and it was the thing that took me longest to get a handle on. A few links:\\n\\n  * Erin Kissane on [frictions people run into when joining Mastodon](https://erinkissane.com/mastodon-is-easy-and-fun-except-when-it-isnt)\\n  * Kyle Kingsbury wrote some great [moderation guidelines for woof.group](https://blog.woof.group/mods/) (note: woof.group is a LGBTQ+ leather instance, be prepared to see lots of NSFW posts if you visit it)\\n  * Mekka Okereke writes [lots of great posts about issues Black people encounter on Mastodon](https://hachyderm.io/@mekkaokereke/110273797004251326) (though they\\'re all on Mastodon so it\\'s a little hard to navigate)\\n\\n\\n\\n### that\\'s all!\\n\\nI don\\'t regret setting up a single-user server - even though it\\'s inconvenient, it\\'s important to me to have control over my social media. I think \"have control over my social media\" is more important to me than it is to most other people though, because I use Twitter/Mastodon a lot for work.\\n\\nI am happy that I didn\\'t _start out_ on a single-user server though - I think it would have made getting started on Mastodon a lot more difficult.\\n\\nMastodon is pretty rough around the edges sometimes but I\\'m able to have more interesting conversations about computers there than I am on Twitter (or Bluesky), so that\\'s where I\\'m staying for now.\\n', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n')"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print(\"Example document:\")\n",
        "documents[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9e8bd061-8ff4-4dd2-922c-117365fd8bdf",
      "metadata": {},
      "source": [
        "# Exercises"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6282009a-3ccd-43fb-92dd-d04383bf4740",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-09-04T17:07:43.349702+0200 - INFO - Building a VectorStoreIndex from 20 documents...\n",
            "2023-09-04T17:08:03.084885+0200 - INFO - Built a VectorStoreIndex from 20 documents\n"
          ]
        }
      ],
      "source": [
        "# Create a vector index from `documents` (this could take a minute because we're processing lots of text)\n",
        "\n",
        "logger.info(f\"Building a VectorStoreIndex from {len(documents)} documents...\")\n",
        "index = llama_index.VectorStoreIndex.from_documents(documents, service_context=service_context)\n",
        "logger.info(f\"Built a VectorStoreIndex from {len(documents)} documents\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "50650136-6ea7-41ed-9b9b-816da3bded7c",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.5832783075911185\n",
            "id_='30daa44b-a604-4aa7-b092-9b11706ef51b' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='7967e750-1049-4eeb-a18d-611b131e6907', node_type=None, metadata={}, hash='60b34270cb3fa7f0974c09e0296fdf2c4bf98715c0b44a4136444b014ef4e031'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='b8240acb-a6c5-46ff-8183-5652117948c7', node_type=None, metadata={}, hash='96e930b94c401d3dfddc7dc30b4068ca399d7944a226f040b7b30bbcc81aeb16'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='d0e34edc-85c2-47ce-aca0-6ab814320d37', node_type=None, metadata={}, hash='c37eb86ecec3c89858b619694fd1fdecd2ad53ebcb69a269cdc49d0eb7158cfa')} hash='9c35a586205d34714de2df12ecbe0f75475bf66b1af46d9776c710d3ccb5f0e1' text=\"Then we'll talk about reading the specification, we'll going to do some experiments, and we're going to implement our own terrible version of DNS.[](https://jvns.ca/images/2022-railsconf/slide-09.png)\\n\\nAnd so what's DNS really briefly?DNS stands for the Domain Name System.And every time you go to a website like `www.example.com`, your browser needs to look up that website's IP address.So DNS translates domain names into IP addresses.It looks up other information about domain names too, but we're mostly just going to talk about IP addresses today.[](https://jvns.ca/images/2022-railsconf/slide-10.png)\\n\\nI want to briefly sell why I think DNS is cool, because we're going to be talking about it a lot.[](https://jvns.ca/images/2022-railsconf/slide-11.png)\\n\\nOne cool thing about DNS is that it's this invisible system that controls the entire internet.\" start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5672421679761838\n",
            "id_='11ad8a43-a957-4803-ac52-5a91085e89df' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='30310330-b558-439d-af05-609c019dc47e', node_type=None, metadata={}, hash='bf74726f1a9d207101585f4bac97887b46f69e144c09d600b271e59d535b7c09'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='52e1c2c8-3623-4dfe-ab6c-c79f534ff3a0', node_type=None, metadata={}, hash='3f5775c93b4c5bf9664252a6217b4f2821d86a79ae9d6cd88ed77ad500051637'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='2787aa31-06a8-4bc0-afd2-667378444dea', node_type=None, metadata={}, hash='8dcc55dd460310a423c55fdf2b5d08d7cbe940a6d558894179804c68f7a6c1ad')} hash='4f6628b79cecbae785dc5de8b837284bc8420fa588546a10441078b6285ba023' text=\"At the time I thought this was silly (I thought DNS parsing was really straightforward and that it was obvious how to do it) but I realized that maybe the commenters didn't think it was easy or obvious, and wanted to know how do it.Which makes sense!It's not obvious!Those comments partly inspired [implement DNS in a weekend](https://implement-dns.wizardzines.com/), which focuses much more heavily on the parsing aspects.\" start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5582626290434468\n",
            "id_='b3dd51cd-6b3f-468d-a56c-102a0f5fd895' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='92d7796e-5c82-4f2a-bdea-47199cb52b07', node_type=None, metadata={}, hash='522e3160df17d868bb935797216652d545bffaf3974ce53dec6c957817a9ab42'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='75f44476-8355-4b94-b43d-0eddc7533276', node_type=None, metadata={}, hash='2a676cd4cef3b5b693fae8e2ab6e164d7554a78c9e0c8cfdfc93d1e232df96c1'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='e1f19b40-c232-4f89-942b-d4196014a1a7', node_type=None, metadata={}, hash='d686ada8e444a65e846dc5c385055b39491c444fa4803ba432f575b1038585e3')} hash='4e141c9b8a20fa329f906668b681f39db5e072d8b7fbd17b6c6700d6278b952e' text=\"I've talked to a lot of smart friends who are very experienced programmers about DNS of the years, and many of them either:\\n\\n  * didn't feel comfortable making simple DNS changes to their websites\\n  * or were confused about basic facts about how DNS works (like that records are [pulled and not pushed](https://jvns.ca/blog/2021/12/06/dns-doesn-t-propagate/))\\n  * or did understand DNS basics pretty well, but had the some of the same knowledge gaps that I'd struggled with (negative caching and the details of how `dig` and your browser do DNS queries differently)\\n\\n\\n\\nSo if we're all struggling with the same things about DNS, what's going on? Why is it so hard to learn for so many people?\\n\\nHere are some ideas.\\n\\n### a lot of the system is hidden\\n\\nWhen you make a DNS request on your computer, the basic story is:\\n\\n  1. your computer makes a request to a server called **resolver**\\n  2. the resolver checks its cache, and makes requests to some other servers called **authoritative nameservers**\" start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5481243282419699\n",
            "id_='dec70a24-56f9-4aef-bca5-0cd488a4b94d' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='92d7796e-5c82-4f2a-bdea-47199cb52b07', node_type=None, metadata={}, hash='522e3160df17d868bb935797216652d545bffaf3974ce53dec6c957817a9ab42'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='7234b8d2-9785-48df-802a-3eb2a133628d', node_type=None, metadata={}, hash='069e1314a2e2f30404eb6b8dafaed901b4d61f0092d4d187395110319320c33c')} hash='c43f4a15b919f9053a3ced4b00521b0debbbdb6756d4840d2b125c9b305418a7' text='One tricky thing about documenting DNS gotchas is that different people are going to run into different gotchas - if you\\'re just configuring DNS for your personal domain once every 3 years, you\\'re probably going to run into different gotchas than someone who administrates DNS for a domain with heavy traffic.\\n\\nA couple of more quick reasons:\\n\\n### infrequent exposure\\n\\nA lot of people only deal with DNS extremely infrequently. And of course if you only touch DNS every 3 years it\\'s going to be harder to learn!\\n\\nI think cheat sheets (like \"here are the steps to changing your nameservers\") can really help with this.\\n\\n### it\\'s hard to experiment with\\n\\nDNS can be scary to experiment with - you don\\'t want to mess up your domain. We built [Mess With DNS](https://messwithdns.net/) to make this one a little easier.\\n\\n### that\\'s all for now\\n\\nI\\'d love to hear other thoughts about what makes DNS (or your favourite mysterious technology) hard to learn.' start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.547887529421278\n",
            "id_='ab87f18a-2e2b-43bd-bb51-8dc08859f6b8' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='92d7796e-5c82-4f2a-bdea-47199cb52b07', node_type=None, metadata={}, hash='522e3160df17d868bb935797216652d545bffaf3974ce53dec6c957817a9ab42'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='e1f19b40-c232-4f89-942b-d4196014a1a7', node_type=None, metadata={}, hash='d686ada8e444a65e846dc5c385055b39491c444fa4803ba432f575b1038585e3'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='882b9b46-72a9-4695-8740-e36af1c13c3f', node_type=None, metadata={}, hash='81548515922615e6da273a31e0340bded5430af6c17790546edeed1ab591fd11')} hash='21a60f80b0206e15f7a4118d4609022ee19a6efe122a6ff547d4a0ed6cf588f7' text='### dealing with hidden systems\\n\\nA couple of ideas for how to deal with hidden systems\\n\\n  * just teaching people what the hidden systems are makes a huge difference. For a long time I had no idea that my computer had many different DNS libraries that were used in different situations and I was confused about this for literally years. This is a big part of my approach.\\n  * with [Mess With DNS](https://messwithdns.net/) we tried out this \"fishbowl\" approach where it shows you some parts of the system (the conversation with the resolver and the authoritative nameserver) that are normally hidden\\n  * I feel like it would be extremely cool to extend DNS to include a \"debugging information\" section. (edit: it looks like this already exists! It\\'s called [Extended DNS Errors](https://blog.nlnetlabs.nl/extended-dns-error-support-for-unbound/), or EDE, and tools are slowly adding support for it.\\n\\n\\n\\n### Extended DNS Errors seem cool\\n\\nExtended DNS Errors are a new way for DNS servers to provide extra debuggging information in DNS responss.' start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5409003903722998\n",
            "id_='02b6a606-7242-43de-801d-7eff2f2cbd36' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='7967e750-1049-4eeb-a18d-611b131e6907', node_type=None, metadata={}, hash='60b34270cb3fa7f0974c09e0296fdf2c4bf98715c0b44a4136444b014ef4e031'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='2c3286a9-830b-4c49-967e-9593bce93599', node_type=None, metadata={}, hash='0d50e56156c3a3566b0173df8dd9488a7934a3d4c9552d9b8967653ed7e760bb'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='f76ec5d1-ff48-4ec1-8e4a-d10d4816f347', node_type=None, metadata={}, hash='5752a3cce15d550b81d51e620716625d92e5a37c1caaef278bc8a2bb4dafeab3')} hash='5e9dbb46787207fd082414145c035fc31f80d93d1e246016dd3dc183609bf153' text='[](https://jvns.ca/images/2022-railsconf/slide-17.png)\\n\\nI\\'ve used example.com instead of maps.google.com on this slide, but the fields are the same.Let\\'s talk about 4 of them: \\n\\nWe have the domain name, no big deal \\n\\nThe Time To Live, which is how long to cache that record for so this is a one day \\n\\nYou have the record type, A stands for address because this is an IP address \\n\\nAnd you have the content, which is the IP address \\n\\n[](https://jvns.ca/images/2022-railsconf/slide-18.png)\\n\\nBut I think that the funniest field in a DNS record is this field in the middle, IN, which stands for INternet.I guess in 1987, they thought that we might be on a lot of different networks.So they made an option for it.In reality, we\\'re all on the internet.And every DNS query has class set to \"internet\".' start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5270712304863349\n",
            "id_='75f44476-8355-4b94-b43d-0eddc7533276' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='92d7796e-5c82-4f2a-bdea-47199cb52b07', node_type=None, metadata={}, hash='522e3160df17d868bb935797216652d545bffaf3974ce53dec6c957817a9ab42'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='30851118-5e4b-4325-ad3d-d566e38fc57c', node_type=None, metadata={}, hash='59ec803793e4f63a40c2efcd164f99e749f38b947761e52d358cdbbe83209570'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='b3dd51cd-6b3f-468d-a56c-102a0f5fd895', node_type=None, metadata={}, hash='4e141c9b8a20fa329f906668b681f39db5e072d8b7fbd17b6c6700d6278b952e')} hash='2a676cd4cef3b5b693fae8e2ab6e164d7554a78c9e0c8cfdfc93d1e232df96c1' text='(I\\'m not going to explain DNS very much in this post, see [Implement DNS in a Weekend](https://implement-dns.wizardzines.com/) or [my DNS blog posts](https://jvns.ca/categories/dns/) for more about how DNS works)\\n\\n### it\\'s not because DNS is super hard\\n\\nWhen I finally learned how to troubleshoot DNS problems, my reaction was \"what, that was it????that\\'s not that hard!\".I felt a little bit cheated!I could explain to you everything that I found confusing about DNS in [a few hours](https://wizardzines.com/zines/dns).So - if DNS is not all that complicated, why did it take me so many years to figure out how to troubleshoot pretty basic DNS issues (like \"my domain doesn\\'t resolve even though I\\'ve set it up correctly\" or \"`dig` and my browser have different DNS results, why?\")?And I wasn\\'t alone in finding DNS hard to learn!' start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5265718853132709\n",
            "id_='81457ae6-1a6a-4e07-ab7c-618986f1c831' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='92d7796e-5c82-4f2a-bdea-47199cb52b07', node_type=None, metadata={}, hash='522e3160df17d868bb935797216652d545bffaf3974ce53dec6c957817a9ab42'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='0e486847-6db0-488e-8dec-c088f80da06a', node_type=None, metadata={}, hash='20e944f7f0b800da05ef00c291f32e272da437eb8e845abc588ae2a68b72238a'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='e060328b-bb15-4ac3-82ad-886b031b40cd', node_type=None, metadata={}, hash='78a856e51f7c30fde0b9717402c1149f6d97176274be0d16b44360cbbd19a625')} hash='8d19cc74896caf1e71ac4ce845d26ad1e726669362349a9bc21bd6ed8949cdab' text=\"(which I talk about in [this talk](https://jvns.ca/blog/2023/05/08/new-talk-learning-dns-in-10-years/)) It took me probably 5 years to realize that I shouldn't visit a domain that doesn't have a DNS record yet, because then the **nonexistance** of that record will be cached, and it gets cached for HOURS, and it's really annoying.\" start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.5248603374700223\n",
            "id_='4bbc8deb-ef9b-4515-bb7c-4e015b93b27c' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='c4ad5dca-4b31-4796-83b7-9dd44cc73e09', node_type=None, metadata={}, hash='ef09bef2b938d8f4e76775043286509b7c1eaad0a2a6816bc7f706435dfb6a0b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='ac539d59-64a3-4cab-856a-3e96a6092c4a', node_type=None, metadata={}, hash='85659e5d13893db65ab8c012171785848eecdeb0577f46f8092506b24bbb040a')} hash='911234fef9dcdb3dc883c0c909e58771898d10744f220449b9fe8d63dd4b3510' text=\"### here's the link again\\n\\nHere's the link to the guide again if you'd like to try it out:\\n\\n>> [Implement DNS in a Weekend](https://implement-dns.wizardzines.com) <<\" start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n",
            "0.4992353629632014\n",
            "id_='7c15897f-bde1-4662-9266-18404289dba8' embedding=None metadata={} excluded_embed_metadata_keys=[] excluded_llm_metadata_keys=[] relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='7967e750-1049-4eeb-a18d-611b131e6907', node_type=None, metadata={}, hash='60b34270cb3fa7f0974c09e0296fdf2c4bf98715c0b44a4136444b014ef4e031'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='e20372bd-0396-44c0-b9ac-a13f064753f3', node_type=None, metadata={}, hash='4b13106f6d3b5f528c8e489dd40620d627d50afc9584b6f891c1951bb51b854e'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='47861e0f-c2e8-467c-ac51-e49693ca5820', node_type=None, metadata={}, hash='72f5d1013cca0711bc3cd83dd4d8af49aedc28632bbd9c0dd883f0eb2e960e55')} hash='7c7b5947bf797ecb08cd8f9aba28d49880d127300ed22fae526de97cc67c4fa4' text=\"[](https://jvns.ca/images/2022-railsconf/slide-51.png)\\n\\nNow I'm going to talk about my favorite strategy for learning about stuff, which is to write my own very bad version of the thing.And I want to say that writing my very bad implementation gives me a really unreasonable amount of confidence.[](https://jvns.ca/images/2022-railsconf/slide-52.png)\\n\\nSo you might think that writing DNS software is complicated, right?But it's easier than you might think, as long as you keep your expectations low.[](https://jvns.ca/images/2022-railsconf/slide-53.png)\\n\\nTo make the DNS queries, the first thing we need to do is we need to make a network connection.Let's do that.[](https://jvns.ca/images/2022-railsconf/slide-54.png)\\n\\nThese four lines of Ruby connect to 8.8.8.8, the Google DNS resolver, on UDP port 53.Now we're like halfway there.\" start_char_idx=None end_char_idx=None text_template='{metadata_str}\\n\\n{content}' metadata_template='{key}: {value}' metadata_seperator='\\n'\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Retrieve blogposts about DNS\n",
        "\n",
        "retriever = index.as_retriever(similarity_top_k=10)\n",
        "results = retriever.retrieve('DNS')\n",
        "\n",
        "for result in results:\n",
        "    print(result.score)\n",
        "    print(result.node)\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e3780a5b-4b23-4c1a-a733-4c0c58998ab7",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Example of a blogpost Document with metadata:\n",
            "{'embedding': None,\n",
            " 'end_char_idx': None,\n",
            " 'excluded_embed_metadata_keys': [],\n",
            " 'excluded_llm_metadata_keys': [],\n",
            " 'hash': '74bb2056380d3e3871dfa6ae4f2b4d9af8e908962b533e58819a94d497caca5b',\n",
            " 'id_': 'Notes on using a single-person Mastodon server',\n",
            " 'metadata': {},\n",
            " 'metadata_seperator': '\\n',\n",
            " 'metadata_template': '{key}: {value}',\n",
            " 'relationships': {},\n",
            " 'start_char_idx': None,\n",
            " 'text': \"I started using Mastodon back in November, and it's the Twitter \"\n",
            "         \"alternative where I've been spending most of my time recently, \"\n",
            "         'mostly because the Fediverse is where a lot of the Linux nerds seem '\n",
            "         'to be right now.\\n'\n",
            "         '\\n'\n",
            "         \"I've found Mastodon quite a bit more confusing than Twitter because \"\n",
            "         \"it's a distributed system, so here are a few technical things I've \"\n",
            "         \"learned about it over the last 10 months. I'll mostly talk about \"\n",
            "         'what using a single-person server has been like for me, as well as a '\n",
            "         'couple of notes about the API, DMs and ActivityPub.\\n'\n",
            "         '\\n'\n",
            "         \"I might have made some mistakes, please let me know if I've gotten \"\n",
            "         'anything wrong!\\n'\n",
            "         '\\n'\n",
            "         \"### what's a mastodon instance?\\n\"\n",
            "         '\\n'\n",
            "         'First: Mastodon is a decentralized collection of independently run '\n",
            "         'servers instead of One Big Server. The software is [open '\n",
            "         'source](https://github.com/mastodon/mastodon).\\n'\n",
            "         '\\n'\n",
            "         'In general, if you have an account on one server (like '\n",
            "         '`ruby.social`), you **can** follow people on another server (like '\n",
            "         '`hachyderm.io`), and they can follow you.\\n'\n",
            "         '\\n'\n",
            "         'I\\'m going to use the terms \"Mastodon server\" and \"Mastodon '\n",
            "         'instance\" interchangeably in this post.\\n'\n",
            "         '\\n'\n",
            "         '### on choosing a Mastodon instance\\n'\n",
            "         '\\n'\n",
            "         'These were the things I was concerned about when choosing an '\n",
            "         'instance:\\n'\n",
            "         '\\n'\n",
            "         '  1. An instance name that I was comfortable being part of my online '\n",
            "         \"identity. For example, I probably wouldn't want to be \"\n",
            "         \"`@b0rk@infosec.exchange` because I'm not an infosec person.\\n\"\n",
            "         \"  2. The server's stability. Most servers are volunteer-run, and \"\n",
            "         'volunteer moderation work can be exhausting - will the server really '\n",
            "         'be around in a few years? For example '\n",
            "         '[mastodon.technology](https://ashfurrow.com/blog/mastodon-technology-shutdown/) '\n",
            "         'and mastodon.lol shut down.\\n'\n",
            "         \"  3. The admins' moderation policies.\\n\"\n",
            "         \"  4. That server's general reputation with other servers. I started \"\n",
            "         'out on `mastodon.social`, but some servers choose to block or limit '\n",
            "         'mastodon.social for various reasons\\n'\n",
            "         '  5. The community: every Mastodon instance has a local timeline '\n",
            "         'with all posts from users on that instance, would I be interested in '\n",
            "         'reading the local timeline?\\n'\n",
            "         '  6. Whether my account would be a burden for the admin of that '\n",
            "         'server (since I have a lot of followers)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'In the end, I chose to run my own mastodon server because it seemed '\n",
            "         \"simplest - I could pick a domain I liked, and I knew I'd definitely \"\n",
            "         \"agree with the moderation decisions because I'd be in charge.\\n\"\n",
            "         '\\n'\n",
            "         \"I'm not going to give server recommendations here, but here's a list \"\n",
            "         'of the [top 200 most common servers people who follow me '\n",
            "         'use](https://gist.github.com/jvns/5eb0a58319b93049a8c014433766edd3).\\n'\n",
            "         '\\n'\n",
            "         '### using your own domain\\n'\n",
            "         '\\n'\n",
            "         'One big thing I wondered was - can I use my own domain (and have the '\n",
            "         \"username `@b0rk@jvns.ca` or something) but be on someone else's \"\n",
            "         'Mastodon server?\\n'\n",
            "         '\\n'\n",
            "         'The answer to this seems to be basically \"no\": if you want to use '\n",
            "         'your own domain on Mastodon, you need to run your own server. (you '\n",
            "         'can [kind of do '\n",
            "         'this](https://blog.maartenballiauw.be/post/2022/11/05/mastodon-own-donain-without-hosting-server.html), '\n",
            "         \"but it's more like an alias or redirect - if I used that method to \"\n",
            "         'direct `b0rk@jvns.ca` to `b0rk@mastodon.social`, my posts would '\n",
            "         'still show up as being from `b0rk@mastodon.social`)\\n'\n",
            "         '\\n'\n",
            "         \"There's also other ActivityPub software \"\n",
            "         '([Takah\u0113](https://jointakahe.org/)) that supports people bringing '\n",
            "         'their own domain in a first-class way.\\n'\n",
            "         '\\n'\n",
            "         '### notes on having my own server\\n'\n",
            "         '\\n'\n",
            "         'I really wanted to have a way to use my own domain name for '\n",
            "         'identity, but to share server hosting costs with other people. This '\n",
            "         \"isn't possible on Mastodon right now, so I decided to set up my own \"\n",
            "         'server instead.\\n'\n",
            "         '\\n'\n",
            "         'I chose to run a Mastodon server (instead of some other ActivityPub '\n",
            "         'implementation) because Mastodon is the most popular one. Good '\n",
            "         'managed Mastodon hosting is readily available, there are tons of '\n",
            "         'options for client apps, and I know for sure that my server will '\n",
            "         \"work well with other people's servers.\\n\"\n",
            "         '\\n'\n",
            "         'I use [masto.host](https://masto.host/) for Mastodon hosting, and '\n",
            "         \"it's been great so far. I have nothing interesting to say about what \"\n",
            "         \"it's like to operate a Mastodon instance because I know literally \"\n",
            "         'nothing about it. Masto.host handles all of the server '\n",
            "         'administration and Mastodon updates, and I never think about it at '\n",
            "         'all.\\n'\n",
            "         '\\n'\n",
            "         'Right now I\\'m on their $19/month (\"Star\") plan, but it\\'s possible '\n",
            "         'I could use a smaller plan with no problems. Right now their '\n",
            "         'cheapest plan is $6/month and I expect that would be fine for '\n",
            "         'someone with a smaller account.\\n'\n",
            "         '\\n'\n",
            "         'Some things I was worried about when embarking on my own Mastodon '\n",
            "         'server:\\n'\n",
            "         '\\n'\n",
            "         '  * I wanted to run the server at `social.jvns.ca`, but I wanted my '\n",
            "         'username to be `b0rk@jvns.ca` instead of `b0rk@social.jvns.ca`. To '\n",
            "         'get this to work I followed these [Setting up a personal fediverse '\n",
            "         'ID](https://jacobian.org/til/my-mastodon-instance/) directions from '\n",
            "         \"Jacob Kaplan-Moss and it's been fine.\\n\"\n",
            "         '  * The administration burden of running my own server. I imported a '\n",
            "         \"small list of servers to block/defederate from but didn't do \"\n",
            "         \"anything else. That's been fine.\\n\"\n",
            "         \"  * Reply and profile visibility. This has been annoying and we'll \"\n",
            "         'talk about it next\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### downsides to being on a single-person server\\n'\n",
            "         '\\n'\n",
            "         'Being on a 1-person server has some significant downsides. To '\n",
            "         'understand why, you need to understand a little about how Mastodon '\n",
            "         'works.\\n'\n",
            "         '\\n'\n",
            "         'Every Mastodon server has a database of posts. Servers only have '\n",
            "         'posts that they were explicitly sent by another server in their '\n",
            "         'database.\\n'\n",
            "         '\\n'\n",
            "         'Some reasons that servers might receive posts:\\n'\n",
            "         '\\n'\n",
            "         '  * someone on the server follows a user\\n'\n",
            "         '  * a post mentions someone on the server\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'As a 1-person server, my server does not receive that many posts! I '\n",
            "         'only get posts from people I follow or posts that explicitly mention '\n",
            "         'me in some way.\\n'\n",
            "         '\\n'\n",
            "         'The causes several problems:\\n'\n",
            "         '\\n'\n",
            "         \"  1. when I visit someone's profile on Mastodon who I don't already \"\n",
            "         \"follow, my server will not fetch the profile's content (it'll fetch \"\n",
            "         'their profile picture, description, and pinned posts, but not any of '\n",
            "         \"their post history). So their profile appears as if they've never \"\n",
            "         'posted anything\\n'\n",
            "         '  2. bad reply visibility: when I look at the replies to somebody '\n",
            "         \"else's post (even if I follow them!), I don't see all of the \"\n",
            "         'replies, only the ones which have made it to my server. If you want '\n",
            "         'to understand the exact rules about who can see which replies (which '\n",
            "         \"are quite complicated!), [here's a great deep \"\n",
            "         'dive](https://seb.jambor.dev/posts/understanding-activitypub-part-3-the-state-of-mastodon/) '\n",
            "         \"by Sebastian Jambor. I think it's possible to end up in a state \"\n",
            "         'where no one person can see all of the replies, including the '\n",
            "         'original poster.\\n'\n",
            "         '  3. favourite and boost accounts are inaccurate - usually posts '\n",
            "         'show up having at most 1 or 2 favourites / boosts, even if the post '\n",
            "         'was actually favourite or boosted hundreds of times. I think this is '\n",
            "         'because it only counts favourites/boosts from people I follow.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'All of these things will happen to users of any small Mastodon '\n",
            "         'server, not just 1-person servers.\\n'\n",
            "         '\\n'\n",
            "         '### bad reply visibility makes conversations harder\\n'\n",
            "         '\\n'\n",
            "         \"A lot of people are on smaller servers, so when they're \"\n",
            "         \"participating in a conversation, they can't see all the replies to \"\n",
            "         'the post.\\n'\n",
            "         '\\n'\n",
            "         'This means that replies can get pretty repetitive because people '\n",
            "         \"literally cannot see each other's replies. This is especially \"\n",
            "         'annoying for posts that are popular or controversial, because the '\n",
            "         'person who made the post has to keep reading similar replies over '\n",
            "         \"and over again by people who think they're making the point for the \"\n",
            "         'first time.\\n'\n",
            "         '\\n'\n",
            "         'To get around this (as a reader), you can click \"open link to post\" '\n",
            "         'or something in your Mastodon client, which will open up the page on '\n",
            "         \"the poster's server where you can read all of the replies. It's \"\n",
            "         'pretty annoying though.\\n'\n",
            "         '\\n'\n",
            "         \"As a poster, I've tried to reduce repetitiveness in replies by:\\n\"\n",
            "         '\\n'\n",
            "         '  * putting requests in my posts like \"(no need to reply if you '\n",
            "         'don\u2019t remember, or if you\u2019ve been using the command line comfortably '\n",
            "         'for 15 years \u2014 this question isn\u2019t for you :) )\"\\n'\n",
            "         '  * occasionally editing my posts to include very common replies\\n'\n",
            "         '  * very occasionally deleting the post if it gets too out of hand\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'The Mastodon devs are extremely aware of these issues, there are a '\n",
            "         'bunch of github issues about them:\\n'\n",
            "         '\\n'\n",
            "         '  * [backfill statuses when first '\n",
            "         'subscribed](https://github.com/mastodon/mastodon/issues/34)\\n'\n",
            "         '  * [fetch whole conversation '\n",
            "         'threads](https://github.com/mastodon/mastodon/issues/9409)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'My guess is that there are technical reasons these features are '\n",
            "         'difficult to add because those issues have been open for 5-7 years.\\n'\n",
            "         '\\n'\n",
            "         'The Mastodon devs have said that they plan to improve reply '\n",
            "         'fetching, but that it requires a significant amount of work.\\n'\n",
            "         '\\n'\n",
            "         '### some visibility workarounds\\n'\n",
            "         '\\n'\n",
            "         'Some people have built workarounds for fetching profiles / replies.\\n'\n",
            "         '\\n'\n",
            "         '  * [Fedifetcher](https://blog.thms.uk/fedifetcher)\\n'\n",
            "         '  * [combine.social](https://combine.social/)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'Also, there are a couple of Mastodon clients which will proactively '\n",
            "         'fetch replies. For iOS:\\n'\n",
            "         '\\n'\n",
            "         '  * Mammoth does it automatically\\n'\n",
            "         '  * Mona will fetch posts if I click \"load from remote server\" '\n",
            "         'manually\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I haven't tried those yet though.\\n\"\n",
            "         '\\n'\n",
            "         '### other downsides of running your own server: discovery is much '\n",
            "         'harder\\n'\n",
            "         '\\n'\n",
            "         'Mastodon instances have a \"local timeline\" where you can see '\n",
            "         'everything other people on the server are posting, and a \"federated '\n",
            "         'timeline\" which shows sort of a combined feed from everyone followed '\n",
            "         'by anyone on the server. This means that you can see trending posts '\n",
            "         \"and get an idea of what's going on and find people to follow. You \"\n",
            "         \"don't get that if you're on a 1-person server - it's just me talking \"\n",
            "         'to myself! (plus occasional interjections from [my reruns '\n",
            "         'bot](https://social.jvns.ca/@b0rk_reruns)).\\n'\n",
            "         '\\n'\n",
            "         'Some workarounds people mentioned for this:\\n'\n",
            "         '\\n'\n",
            "         '  * you can populate your federated timeline with posts from another '\n",
            "         \"instance by using a **relay**. I haven't done this but someone else \"\n",
            "         'said they use [FediBuzz](https://relay.fedi.buzz) and I might try it '\n",
            "         'out.\\n'\n",
            "         '  * some mastodon clients (like apparently Moshidon on Android) let '\n",
            "         'you follow other instances\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'If anyone else on small servers has suggestions for how to make '\n",
            "         \"discovery easier I'd love to hear them.\\n\"\n",
            "         '\\n'\n",
            "         '### account migration\\n'\n",
            "         '\\n'\n",
            "         'When I moved to my own server from `mastodon.social`, I needed to '\n",
            "         \"run an account migration to move over my followers. First, here's \"\n",
            "         'how migration works:\\n'\n",
            "         '\\n'\n",
            "         '  1. Account migration **does not** move over your posts. All of my '\n",
            "         'posts stayed on my old account. This is part of why I moved to '\n",
            "         \"running my own server - I didn't want to ever lose my posts a second \"\n",
            "         'time.\\n'\n",
            "         '  2. Account migration **does not** move over the list of people you '\n",
            "         'follow/mute/block. But you can import/export that list in your '\n",
            "         \"Mastodon settings so it's not a big deal. If you follow private \"\n",
            "         \"accounts they'll have to re-approve your follow request.\\n\"\n",
            "         '  3. Account migration **does** move over your followers\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"The follower move was the part I was most worried about. Here's how \"\n",
            "         'it turned out:\\n'\n",
            "         '\\n'\n",
            "         '  * over ~24 hours, most of my followers moved to the new account\\n'\n",
            "         '  * one or two servers did not get the message about the account '\n",
            "         'migration for some reason, so about 2000 followers were \"stuck\" and '\n",
            "         \"didn't migrate. I fixed this by waiting 30 days and re-running the \"\n",
            "         'account migration, which moved over most of the remaining followers. '\n",
            "         \"There's also a [tootctl \"\n",
            "         'command](https://github.com/mastodon/mastodon/issues/22281) that the '\n",
            "         'admin of the **old instance** can run to retry the migration\\n'\n",
            "         '  * about 200 of my followers never migrated over, I think because '\n",
            "         \"they're using ActivityPub software other than Mastodon which doesn't \"\n",
            "         'support account migration. You can see the [old account '\n",
            "         'here](https://mastodon.social/@b0rk)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### using the Mastodon API is great\\n'\n",
            "         '\\n'\n",
            "         \"One thing I love about Mastodon is - it has an API that's MUCH \"\n",
            "         \"easier to use than Twitter's API. I've always been frustrated with \"\n",
            "         'how difficult it is to navigate large Twitter threads, so I made a '\n",
            "         'small [mastodon thread view](https://mastodon-thread-view.jvns.ca/) '\n",
            "         \"website that lets you log into your Mastodon account. It's pretty \"\n",
            "         \"janky and it's really only made for me to use, but I've really \"\n",
            "         'appreciated the ability to write my own janky software to improve my '\n",
            "         'Mastodon experience.\\n'\n",
            "         '\\n'\n",
            "         'Some notes on the Mastodon API:\\n'\n",
            "         '\\n'\n",
            "         '  * You can build Mastodon client software totally on the frontend '\n",
            "         'in Javascript, which is really cool.\\n'\n",
            "         \"  * I couldn't find a vanilla Javascript Mastodon client, so I \"\n",
            "         '[wrote a crappy '\n",
            "         'one](https://github.com/jvns/mastodon-threaded-replies/blob/main/mastodon.js)\\n'\n",
            "         '  * [API docs are '\n",
            "         'here](https://docs.joinmastodon.org/client/intro/)\\n'\n",
            "         \"  * Here's a [tiny Python script I used to list all my Mastodon \"\n",
            "         'followers](https://gist.github.com/jvns/0fe51383cbbb63e94177c60f1e0371c6), '\n",
            "         'which also serves as a simple example of how easy using the API is.\\n'\n",
            "         '  * The best documentation I could find for which OAuth scopes '\n",
            "         'correspond to which API endpoints is [this github '\n",
            "         'issue](https://github.com/mastodon/mastodon/pull/7929)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"Next I'll talk about a few general things about Mastodon that \"\n",
            "         \"confused or surprised me that aren't specific to being on a \"\n",
            "         'single-person instance.\\n'\n",
            "         '\\n'\n",
            "         '### DMs are weird\\n'\n",
            "         '\\n'\n",
            "         'The way Mastodon DMs work surprised me in a few ways:\\n'\n",
            "         '\\n'\n",
            "         '  * Technically DMs are just regular posts with visibility limited '\n",
            "         'to the people mentioned in the post. This means that if you '\n",
            "         'accidentally mention someone in a DM (\"@x is such a jerk\"), it\\'s '\n",
            "         'possible to accidentally send the message to them\\n'\n",
            "         \"  * DMs aren't very private: the admins on the sending and receiving \"\n",
            "         'servers can technically read your DMs if they have access to the '\n",
            "         \"database. So they're not appropriate for sensitive information.\\n\"\n",
            "         \"  * Turning off DMs is weird. Personally I don't like receiving DMs \"\n",
            "         \"from strangers - it's too much to keep track of and I'd prefer that \"\n",
            "         \"people email me. On Twitter, I can just turn it off and people won't \"\n",
            "         'see an option to DM me. But on Mastodon, when I turn off '\n",
            "         'notifications for DMs, anyone can still \"DM\" me, but the message '\n",
            "         \"will go into a black hole and I'll never see it. I put a note in my \"\n",
            "         'profile about this.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### defederation and limiting\\n'\n",
            "         '\\n'\n",
            "         'There are a couple of different ways for a server to block another '\n",
            "         \"Mastodon server. I haven't really had to do this much but people \"\n",
            "         'talk about it a lot and I was confused about the difference, so:\\n'\n",
            "         '\\n'\n",
            "         '  * A server can **defederate** from another server (this seems to '\n",
            "         'be called '\n",
            "         '[suspend](https://docs.joinmastodon.org/admin/moderation/#suspend-server) '\n",
            "         'in the Mastodon docs). This means that nobody on a server can follow '\n",
            "         'someone from the other server.\\n'\n",
            "         '  * A server can '\n",
            "         '**[limit](https://docs.joinmastodon.org/admin/moderation/#limit-user)** '\n",
            "         '(also known as \"silence\") a user or server. This means that content '\n",
            "         \"from that user is only visible to that user's followers - people \"\n",
            "         'can\\'t discover the user through retweets (aka \"boosts\" on '\n",
            "         'Mastodon).\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"One thing that wasn't obvious to me is that who servers defederate / \"\n",
            "         \"limit is sometimes hidden, so it's hard to suss out what's going on \"\n",
            "         \"if you're considering joining a server, or trying to understand why \"\n",
            "         \"you can't see certain posts.\\n\"\n",
            "         '\\n'\n",
            "         \"### there's no search for posts\\n\"\n",
            "         '\\n'\n",
            "         \"There's no way to search past posts you've read. If I see something \"\n",
            "         'interesting on my timeline and want to find it later, I usually '\n",
            "         \"can't. (Mastodon has a [Elasticsearch-based search \"\n",
            "         'feature](https://docs.joinmastodon.org/admin/optional/elasticsearch/), '\n",
            "         'but it only allows you to search your own posts, your mentions, your '\n",
            "         'favourites, and your bookmarks)\\n'\n",
            "         '\\n'\n",
            "         'These limitations on search are intentional (and a very common '\n",
            "         \"source of arguments) - it's a privacy / safety issue. Here's a \"\n",
            "         '[summary from Tim '\n",
            "         'Bray](https://www.tbray.org/ongoing/When/202x/2022/12/30/Mastodon-Privacy-and-Search) '\n",
            "         'with lots of links.\\n'\n",
            "         '\\n'\n",
            "         'It would be personally convenient for me to be able to search more '\n",
            "         \"easily but I respect folks' safety concerns so I'll leave it at \"\n",
            "         'that.\\n'\n",
            "         '\\n'\n",
            "         'My understanding is that the Mastodon devs are planning to add '\n",
            "         'opt-in search for public posts relatively soon.\\n'\n",
            "         '\\n'\n",
            "         '### other ActivityPub software\\n'\n",
            "         '\\n'\n",
            "         \"We've been talking about Mastodon a lot, but not everyone who I \"\n",
            "         'follow is using Mastodon: Mastodon uses a protocol called '\n",
            "         '[ActivityPub](https://activitypub.rocks/) to distribute messages.\\n'\n",
            "         '\\n'\n",
            "         'Here are some examples of other software I see people talking about, '\n",
            "         'in no particular order:\\n'\n",
            "         '\\n'\n",
            "         '  * [Calckey](https://calckey.org/)\\n'\n",
            "         '  * [Akkoma](https://akkoma.social/)\\n'\n",
            "         '  * [gotosocial](https://gotosocial.org/)\\n'\n",
            "         '  * [Takah\u0113](https://jointakahe.org/)\\n'\n",
            "         '  * [writefreely](https://writefreely.org/)\\n'\n",
            "         '  * [pixelfed](https://pixelfed.org/) (for images)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I'm probably missing a bunch of important ones.\\n\"\n",
            "         '\\n'\n",
            "         \"### what's the difference between Mastodon and other ActivityPub \"\n",
            "         'software?\\n'\n",
            "         '\\n'\n",
            "         \"This confused me for a while, and I'm still not super clear on how \"\n",
            "         \"ActivityPub works. What I've understood is:\\n\"\n",
            "         '\\n'\n",
            "         '  * ActivityPub is a protocol (you can explore how it works with '\n",
            "         \"blinry's nice [JSON \"\n",
            "         'explorer](https://json.blinry.org/#https://chaos.social/users/blinry))\\n'\n",
            "         '  * Mastodon **servers** communicate with each other (and with other '\n",
            "         'ActivityPub servers) using ActivityPub\\n'\n",
            "         '  * Mastodon **clients** communicate with their server using the '\n",
            "         'Mastodon API, which is its own thing\\n'\n",
            "         \"  * There's also software like \"\n",
            "         '[GoToSocial](https://github.com/superseriousbusiness/gotosocial) '\n",
            "         'that aims to be compatible with the Mastodon API, so that you can '\n",
            "         'use a Mastodon client with it\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### more mastodon resources\\n'\n",
            "         '\\n'\n",
            "         '  * [Fedi.Tips](https://fedi.tips/) seems to be a great '\n",
            "         'introduction\\n'\n",
            "         '  * I think you can still use '\n",
            "         '[FediFinder](https://fedifinder.glitch.me/) to find folks you '\n",
            "         'followed on Twitter on Mastodon\\n'\n",
            "         \"  * I've been using the [Ivory](https://tapbots.com/ivory/) client \"\n",
            "         'on iOS, but there are lots of great clients. '\n",
            "         '[Elk](https://elk.zone/) is an alternative web client that folks '\n",
            "         'seem to like.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I haven't written here about what Mastodon culture is like because \"\n",
            "         'other people have done a much better job of talking about it than '\n",
            "         \"me, but of course it's is the biggest thing that affects your \"\n",
            "         'experience and it was the thing that took me longest to get a handle '\n",
            "         'on. A few links:\\n'\n",
            "         '\\n'\n",
            "         '  * Erin Kissane on [frictions people run into when joining '\n",
            "         'Mastodon](https://erinkissane.com/mastodon-is-easy-and-fun-except-when-it-isnt)\\n'\n",
            "         '  * Kyle Kingsbury wrote some great [moderation guidelines for '\n",
            "         'woof.group](https://blog.woof.group/mods/) (note: woof.group is a '\n",
            "         'LGBTQ+ leather instance, be prepared to see lots of NSFW posts if '\n",
            "         'you visit it)\\n'\n",
            "         '  * Mekka Okereke writes [lots of great posts about issues Black '\n",
            "         'people encounter on '\n",
            "         'Mastodon](https://hachyderm.io/@mekkaokereke/110273797004251326) '\n",
            "         \"(though they're all on Mastodon so it's a little hard to navigate)\\n\"\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"### that's all!\\n\"\n",
            "         '\\n'\n",
            "         \"I don't regret setting up a single-user server - even though it's \"\n",
            "         \"inconvenient, it's important to me to have control over my social \"\n",
            "         'media. I think \"have control over my social media\" is more important '\n",
            "         'to me than it is to most other people though, because I use '\n",
            "         'Twitter/Mastodon a lot for work.\\n'\n",
            "         '\\n'\n",
            "         \"I am happy that I didn't _start out_ on a single-user server though \"\n",
            "         '- I think it would have made getting started on Mastodon a lot more '\n",
            "         'difficult.\\n'\n",
            "         '\\n'\n",
            "         \"Mastodon is pretty rough around the edges sometimes but I'm able to \"\n",
            "         'have more interesting conversations about computers there than I am '\n",
            "         \"on Twitter (or Bluesky), so that's where I'm staying for now.\\n\",\n",
            " 'text_template': '{metadata_str}\\n\\n{content}'}\n"
          ]
        }
      ],
      "source": [
        "# It's really hard to figure out from which blogposts these results are! Create a set of Documents that has the blogpost title as the id.\n",
        "\n",
        "documents = []\n",
        "for blogpost in blogposts:\n",
        "    document = llama_index.Document(text=blogpost['text'], id_= blogpost['title'])\n",
        "    documents.append(document)\n",
        "\n",
        "print(\"Example of a blogpost Document with metadata:\")\n",
        "pprint.pprint(dict(documents[0]))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8f68d2b6-23ea-4a9d-bc9c-87743dbec032",
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "47688e4b-56bb-40a5-83dd-6e98002ad2ac",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Build a vector index from these new documents\n",
        "\n",
        "index = llama_index.VectorStoreIndex.from_documents(documents, service_context=service_context)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e4f577c5-98af-4a3d-8fc4-324a2ab0bc1a",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.5832783075911185\n",
            "New talk: Learning DNS in 10 years\n",
            "\n",
            "0.5672421679761838\n",
            "Some blogging myths\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Retrieve blogposts about DNS using the new index. This time, use the id of the source node to check from which blogpost they originate.\n",
        "\n",
        "retriever = index.as_retriever()\n",
        "results = retriever.retrieve('DNS')\n",
        "\n",
        "for result in results:\n",
        "    print(result.score)\n",
        "    print(result.node.source_node.node_id)\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f4b91d79-a54f-4f1a-aaf3-8ce1c01dbc8d",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-09-04T17:08:23.621461+0200 - INFO - Saved VectorStoreIndex to indices/julia_evans_blogposts\n"
          ]
        }
      ],
      "source": [
        "# Store your index to disk\n",
        "\n",
        "index_path = 'indices/julia_evans_blogposts'\n",
        "index.storage_context.persist(index_path)\n",
        "logger.info(f\"Saved VectorStoreIndex to {index_path}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dbd7f948-cb5d-49be-a505-86e27e915ea9",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "LLM is explicitly disabled. Using MockLLM.\n"
          ]
        }
      ],
      "source": [
        "# Load your index from disk in a new index variable. Hint: you need two ingredients: a storage context and a service context.\n",
        "\n",
        "index_path = 'indices/julia_evans_blogposts'\n",
        "storage_context = llama_index.StorageContext.from_defaults(persist_dir=index_path)\n",
        "service_context = llama_index.ServiceContext.from_defaults(embed_model=\"local:sentence-transformers/all-minilm-l6-v2\", chunk_size=256, llm=None)\n",
        "index2 = llama_index.load_index_from_storage(storage_context, service_context=service_context)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dd76ae61-63bb-4ae3-af27-765209ddc4a1",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Expand the documents in your index with metadata such as title, original URL, author, and update time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9faf9c3c-cfba-486c-a6fa-88911132ebe6",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'author': 'Julia Evans',\n",
            " 'guidislink': False,\n",
            " 'id': 'https://jvns.ca/blog/2023/08/11/some-notes-on-mastodon/',\n",
            " 'link': 'https://jvns.ca/blog/2023/08/11/some-notes-on-mastodon/',\n",
            " 'text': \"I started using Mastodon back in November, and it's the Twitter \"\n",
            "         \"alternative where I've been spending most of my time recently, \"\n",
            "         'mostly because the Fediverse is where a lot of the Linux nerds seem '\n",
            "         'to be right now.\\n'\n",
            "         '\\n'\n",
            "         \"I've found Mastodon quite a bit more confusing than Twitter because \"\n",
            "         \"it's a distributed system, so here are a few technical things I've \"\n",
            "         \"learned about it over the last 10 months. I'll mostly talk about \"\n",
            "         'what using a single-person server has been like for me, as well as a '\n",
            "         'couple of notes about the API, DMs and ActivityPub.\\n'\n",
            "         '\\n'\n",
            "         \"I might have made some mistakes, please let me know if I've gotten \"\n",
            "         'anything wrong!\\n'\n",
            "         '\\n'\n",
            "         \"### what's a mastodon instance?\\n\"\n",
            "         '\\n'\n",
            "         'First: Mastodon is a decentralized collection of independently run '\n",
            "         'servers instead of One Big Server. The software is [open '\n",
            "         'source](https://github.com/mastodon/mastodon).\\n'\n",
            "         '\\n'\n",
            "         'In general, if you have an account on one server (like '\n",
            "         '`ruby.social`), you **can** follow people on another server (like '\n",
            "         '`hachyderm.io`), and they can follow you.\\n'\n",
            "         '\\n'\n",
            "         'I\\'m going to use the terms \"Mastodon server\" and \"Mastodon '\n",
            "         'instance\" interchangeably in this post.\\n'\n",
            "         '\\n'\n",
            "         '### on choosing a Mastodon instance\\n'\n",
            "         '\\n'\n",
            "         'These were the things I was concerned about when choosing an '\n",
            "         'instance:\\n'\n",
            "         '\\n'\n",
            "         '  1. An instance name that I was comfortable being part of my online '\n",
            "         \"identity. For example, I probably wouldn't want to be \"\n",
            "         \"`@b0rk@infosec.exchange` because I'm not an infosec person.\\n\"\n",
            "         \"  2. The server's stability. Most servers are volunteer-run, and \"\n",
            "         'volunteer moderation work can be exhausting - will the server really '\n",
            "         'be around in a few years? For example '\n",
            "         '[mastodon.technology](https://ashfurrow.com/blog/mastodon-technology-shutdown/) '\n",
            "         'and mastodon.lol shut down.\\n'\n",
            "         \"  3. The admins' moderation policies.\\n\"\n",
            "         \"  4. That server's general reputation with other servers. I started \"\n",
            "         'out on `mastodon.social`, but some servers choose to block or limit '\n",
            "         'mastodon.social for various reasons\\n'\n",
            "         '  5. The community: every Mastodon instance has a local timeline '\n",
            "         'with all posts from users on that instance, would I be interested in '\n",
            "         'reading the local timeline?\\n'\n",
            "         '  6. Whether my account would be a burden for the admin of that '\n",
            "         'server (since I have a lot of followers)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'In the end, I chose to run my own mastodon server because it seemed '\n",
            "         \"simplest - I could pick a domain I liked, and I knew I'd definitely \"\n",
            "         \"agree with the moderation decisions because I'd be in charge.\\n\"\n",
            "         '\\n'\n",
            "         \"I'm not going to give server recommendations here, but here's a list \"\n",
            "         'of the [top 200 most common servers people who follow me '\n",
            "         'use](https://gist.github.com/jvns/5eb0a58319b93049a8c014433766edd3).\\n'\n",
            "         '\\n'\n",
            "         '### using your own domain\\n'\n",
            "         '\\n'\n",
            "         'One big thing I wondered was - can I use my own domain (and have the '\n",
            "         \"username `@b0rk@jvns.ca` or something) but be on someone else's \"\n",
            "         'Mastodon server?\\n'\n",
            "         '\\n'\n",
            "         'The answer to this seems to be basically \"no\": if you want to use '\n",
            "         'your own domain on Mastodon, you need to run your own server. (you '\n",
            "         'can [kind of do '\n",
            "         'this](https://blog.maartenballiauw.be/post/2022/11/05/mastodon-own-donain-without-hosting-server.html), '\n",
            "         \"but it's more like an alias or redirect - if I used that method to \"\n",
            "         'direct `b0rk@jvns.ca` to `b0rk@mastodon.social`, my posts would '\n",
            "         'still show up as being from `b0rk@mastodon.social`)\\n'\n",
            "         '\\n'\n",
            "         \"There's also other ActivityPub software \"\n",
            "         '([Takah\u0113](https://jointakahe.org/)) that supports people bringing '\n",
            "         'their own domain in a first-class way.\\n'\n",
            "         '\\n'\n",
            "         '### notes on having my own server\\n'\n",
            "         '\\n'\n",
            "         'I really wanted to have a way to use my own domain name for '\n",
            "         'identity, but to share server hosting costs with other people. This '\n",
            "         \"isn't possible on Mastodon right now, so I decided to set up my own \"\n",
            "         'server instead.\\n'\n",
            "         '\\n'\n",
            "         'I chose to run a Mastodon server (instead of some other ActivityPub '\n",
            "         'implementation) because Mastodon is the most popular one. Good '\n",
            "         'managed Mastodon hosting is readily available, there are tons of '\n",
            "         'options for client apps, and I know for sure that my server will '\n",
            "         \"work well with other people's servers.\\n\"\n",
            "         '\\n'\n",
            "         'I use [masto.host](https://masto.host/) for Mastodon hosting, and '\n",
            "         \"it's been great so far. I have nothing interesting to say about what \"\n",
            "         \"it's like to operate a Mastodon instance because I know literally \"\n",
            "         'nothing about it. Masto.host handles all of the server '\n",
            "         'administration and Mastodon updates, and I never think about it at '\n",
            "         'all.\\n'\n",
            "         '\\n'\n",
            "         'Right now I\\'m on their $19/month (\"Star\") plan, but it\\'s possible '\n",
            "         'I could use a smaller plan with no problems. Right now their '\n",
            "         'cheapest plan is $6/month and I expect that would be fine for '\n",
            "         'someone with a smaller account.\\n'\n",
            "         '\\n'\n",
            "         'Some things I was worried about when embarking on my own Mastodon '\n",
            "         'server:\\n'\n",
            "         '\\n'\n",
            "         '  * I wanted to run the server at `social.jvns.ca`, but I wanted my '\n",
            "         'username to be `b0rk@jvns.ca` instead of `b0rk@social.jvns.ca`. To '\n",
            "         'get this to work I followed these [Setting up a personal fediverse '\n",
            "         'ID](https://jacobian.org/til/my-mastodon-instance/) directions from '\n",
            "         \"Jacob Kaplan-Moss and it's been fine.\\n\"\n",
            "         '  * The administration burden of running my own server. I imported a '\n",
            "         \"small list of servers to block/defederate from but didn't do \"\n",
            "         \"anything else. That's been fine.\\n\"\n",
            "         \"  * Reply and profile visibility. This has been annoying and we'll \"\n",
            "         'talk about it next\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### downsides to being on a single-person server\\n'\n",
            "         '\\n'\n",
            "         'Being on a 1-person server has some significant downsides. To '\n",
            "         'understand why, you need to understand a little about how Mastodon '\n",
            "         'works.\\n'\n",
            "         '\\n'\n",
            "         'Every Mastodon server has a database of posts. Servers only have '\n",
            "         'posts that they were explicitly sent by another server in their '\n",
            "         'database.\\n'\n",
            "         '\\n'\n",
            "         'Some reasons that servers might receive posts:\\n'\n",
            "         '\\n'\n",
            "         '  * someone on the server follows a user\\n'\n",
            "         '  * a post mentions someone on the server\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'As a 1-person server, my server does not receive that many posts! I '\n",
            "         'only get posts from people I follow or posts that explicitly mention '\n",
            "         'me in some way.\\n'\n",
            "         '\\n'\n",
            "         'The causes several problems:\\n'\n",
            "         '\\n'\n",
            "         \"  1. when I visit someone's profile on Mastodon who I don't already \"\n",
            "         \"follow, my server will not fetch the profile's content (it'll fetch \"\n",
            "         'their profile picture, description, and pinned posts, but not any of '\n",
            "         \"their post history). So their profile appears as if they've never \"\n",
            "         'posted anything\\n'\n",
            "         '  2. bad reply visibility: when I look at the replies to somebody '\n",
            "         \"else's post (even if I follow them!), I don't see all of the \"\n",
            "         'replies, only the ones which have made it to my server. If you want '\n",
            "         'to understand the exact rules about who can see which replies (which '\n",
            "         \"are quite complicated!), [here's a great deep \"\n",
            "         'dive](https://seb.jambor.dev/posts/understanding-activitypub-part-3-the-state-of-mastodon/) '\n",
            "         \"by Sebastian Jambor. I think it's possible to end up in a state \"\n",
            "         'where no one person can see all of the replies, including the '\n",
            "         'original poster.\\n'\n",
            "         '  3. favourite and boost accounts are inaccurate - usually posts '\n",
            "         'show up having at most 1 or 2 favourites / boosts, even if the post '\n",
            "         'was actually favourite or boosted hundreds of times. I think this is '\n",
            "         'because it only counts favourites/boosts from people I follow.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'All of these things will happen to users of any small Mastodon '\n",
            "         'server, not just 1-person servers.\\n'\n",
            "         '\\n'\n",
            "         '### bad reply visibility makes conversations harder\\n'\n",
            "         '\\n'\n",
            "         \"A lot of people are on smaller servers, so when they're \"\n",
            "         \"participating in a conversation, they can't see all the replies to \"\n",
            "         'the post.\\n'\n",
            "         '\\n'\n",
            "         'This means that replies can get pretty repetitive because people '\n",
            "         \"literally cannot see each other's replies. This is especially \"\n",
            "         'annoying for posts that are popular or controversial, because the '\n",
            "         'person who made the post has to keep reading similar replies over '\n",
            "         \"and over again by people who think they're making the point for the \"\n",
            "         'first time.\\n'\n",
            "         '\\n'\n",
            "         'To get around this (as a reader), you can click \"open link to post\" '\n",
            "         'or something in your Mastodon client, which will open up the page on '\n",
            "         \"the poster's server where you can read all of the replies. It's \"\n",
            "         'pretty annoying though.\\n'\n",
            "         '\\n'\n",
            "         \"As a poster, I've tried to reduce repetitiveness in replies by:\\n\"\n",
            "         '\\n'\n",
            "         '  * putting requests in my posts like \"(no need to reply if you '\n",
            "         'don\u2019t remember, or if you\u2019ve been using the command line comfortably '\n",
            "         'for 15 years \u2014 this question isn\u2019t for you :) )\"\\n'\n",
            "         '  * occasionally editing my posts to include very common replies\\n'\n",
            "         '  * very occasionally deleting the post if it gets too out of hand\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'The Mastodon devs are extremely aware of these issues, there are a '\n",
            "         'bunch of github issues about them:\\n'\n",
            "         '\\n'\n",
            "         '  * [backfill statuses when first '\n",
            "         'subscribed](https://github.com/mastodon/mastodon/issues/34)\\n'\n",
            "         '  * [fetch whole conversation '\n",
            "         'threads](https://github.com/mastodon/mastodon/issues/9409)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'My guess is that there are technical reasons these features are '\n",
            "         'difficult to add because those issues have been open for 5-7 years.\\n'\n",
            "         '\\n'\n",
            "         'The Mastodon devs have said that they plan to improve reply '\n",
            "         'fetching, but that it requires a significant amount of work.\\n'\n",
            "         '\\n'\n",
            "         '### some visibility workarounds\\n'\n",
            "         '\\n'\n",
            "         'Some people have built workarounds for fetching profiles / replies.\\n'\n",
            "         '\\n'\n",
            "         '  * [Fedifetcher](https://blog.thms.uk/fedifetcher)\\n'\n",
            "         '  * [combine.social](https://combine.social/)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'Also, there are a couple of Mastodon clients which will proactively '\n",
            "         'fetch replies. For iOS:\\n'\n",
            "         '\\n'\n",
            "         '  * Mammoth does it automatically\\n'\n",
            "         '  * Mona will fetch posts if I click \"load from remote server\" '\n",
            "         'manually\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I haven't tried those yet though.\\n\"\n",
            "         '\\n'\n",
            "         '### other downsides of running your own server: discovery is much '\n",
            "         'harder\\n'\n",
            "         '\\n'\n",
            "         'Mastodon instances have a \"local timeline\" where you can see '\n",
            "         'everything other people on the server are posting, and a \"federated '\n",
            "         'timeline\" which shows sort of a combined feed from everyone followed '\n",
            "         'by anyone on the server. This means that you can see trending posts '\n",
            "         \"and get an idea of what's going on and find people to follow. You \"\n",
            "         \"don't get that if you're on a 1-person server - it's just me talking \"\n",
            "         'to myself! (plus occasional interjections from [my reruns '\n",
            "         'bot](https://social.jvns.ca/@b0rk_reruns)).\\n'\n",
            "         '\\n'\n",
            "         'Some workarounds people mentioned for this:\\n'\n",
            "         '\\n'\n",
            "         '  * you can populate your federated timeline with posts from another '\n",
            "         \"instance by using a **relay**. I haven't done this but someone else \"\n",
            "         'said they use [FediBuzz](https://relay.fedi.buzz) and I might try it '\n",
            "         'out.\\n'\n",
            "         '  * some mastodon clients (like apparently Moshidon on Android) let '\n",
            "         'you follow other instances\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         'If anyone else on small servers has suggestions for how to make '\n",
            "         \"discovery easier I'd love to hear them.\\n\"\n",
            "         '\\n'\n",
            "         '### account migration\\n'\n",
            "         '\\n'\n",
            "         'When I moved to my own server from `mastodon.social`, I needed to '\n",
            "         \"run an account migration to move over my followers. First, here's \"\n",
            "         'how migration works:\\n'\n",
            "         '\\n'\n",
            "         '  1. Account migration **does not** move over your posts. All of my '\n",
            "         'posts stayed on my old account. This is part of why I moved to '\n",
            "         \"running my own server - I didn't want to ever lose my posts a second \"\n",
            "         'time.\\n'\n",
            "         '  2. Account migration **does not** move over the list of people you '\n",
            "         'follow/mute/block. But you can import/export that list in your '\n",
            "         \"Mastodon settings so it's not a big deal. If you follow private \"\n",
            "         \"accounts they'll have to re-approve your follow request.\\n\"\n",
            "         '  3. Account migration **does** move over your followers\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"The follower move was the part I was most worried about. Here's how \"\n",
            "         'it turned out:\\n'\n",
            "         '\\n'\n",
            "         '  * over ~24 hours, most of my followers moved to the new account\\n'\n",
            "         '  * one or two servers did not get the message about the account '\n",
            "         'migration for some reason, so about 2000 followers were \"stuck\" and '\n",
            "         \"didn't migrate. I fixed this by waiting 30 days and re-running the \"\n",
            "         'account migration, which moved over most of the remaining followers. '\n",
            "         \"There's also a [tootctl \"\n",
            "         'command](https://github.com/mastodon/mastodon/issues/22281) that the '\n",
            "         'admin of the **old instance** can run to retry the migration\\n'\n",
            "         '  * about 200 of my followers never migrated over, I think because '\n",
            "         \"they're using ActivityPub software other than Mastodon which doesn't \"\n",
            "         'support account migration. You can see the [old account '\n",
            "         'here](https://mastodon.social/@b0rk)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### using the Mastodon API is great\\n'\n",
            "         '\\n'\n",
            "         \"One thing I love about Mastodon is - it has an API that's MUCH \"\n",
            "         \"easier to use than Twitter's API. I've always been frustrated with \"\n",
            "         'how difficult it is to navigate large Twitter threads, so I made a '\n",
            "         'small [mastodon thread view](https://mastodon-thread-view.jvns.ca/) '\n",
            "         \"website that lets you log into your Mastodon account. It's pretty \"\n",
            "         \"janky and it's really only made for me to use, but I've really \"\n",
            "         'appreciated the ability to write my own janky software to improve my '\n",
            "         'Mastodon experience.\\n'\n",
            "         '\\n'\n",
            "         'Some notes on the Mastodon API:\\n'\n",
            "         '\\n'\n",
            "         '  * You can build Mastodon client software totally on the frontend '\n",
            "         'in Javascript, which is really cool.\\n'\n",
            "         \"  * I couldn't find a vanilla Javascript Mastodon client, so I \"\n",
            "         '[wrote a crappy '\n",
            "         'one](https://github.com/jvns/mastodon-threaded-replies/blob/main/mastodon.js)\\n'\n",
            "         '  * [API docs are '\n",
            "         'here](https://docs.joinmastodon.org/client/intro/)\\n'\n",
            "         \"  * Here's a [tiny Python script I used to list all my Mastodon \"\n",
            "         'followers](https://gist.github.com/jvns/0fe51383cbbb63e94177c60f1e0371c6), '\n",
            "         'which also serves as a simple example of how easy using the API is.\\n'\n",
            "         '  * The best documentation I could find for which OAuth scopes '\n",
            "         'correspond to which API endpoints is [this github '\n",
            "         'issue](https://github.com/mastodon/mastodon/pull/7929)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"Next I'll talk about a few general things about Mastodon that \"\n",
            "         \"confused or surprised me that aren't specific to being on a \"\n",
            "         'single-person instance.\\n'\n",
            "         '\\n'\n",
            "         '### DMs are weird\\n'\n",
            "         '\\n'\n",
            "         'The way Mastodon DMs work surprised me in a few ways:\\n'\n",
            "         '\\n'\n",
            "         '  * Technically DMs are just regular posts with visibility limited '\n",
            "         'to the people mentioned in the post. This means that if you '\n",
            "         'accidentally mention someone in a DM (\"@x is such a jerk\"), it\\'s '\n",
            "         'possible to accidentally send the message to them\\n'\n",
            "         \"  * DMs aren't very private: the admins on the sending and receiving \"\n",
            "         'servers can technically read your DMs if they have access to the '\n",
            "         \"database. So they're not appropriate for sensitive information.\\n\"\n",
            "         \"  * Turning off DMs is weird. Personally I don't like receiving DMs \"\n",
            "         \"from strangers - it's too much to keep track of and I'd prefer that \"\n",
            "         \"people email me. On Twitter, I can just turn it off and people won't \"\n",
            "         'see an option to DM me. But on Mastodon, when I turn off '\n",
            "         'notifications for DMs, anyone can still \"DM\" me, but the message '\n",
            "         \"will go into a black hole and I'll never see it. I put a note in my \"\n",
            "         'profile about this.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### defederation and limiting\\n'\n",
            "         '\\n'\n",
            "         'There are a couple of different ways for a server to block another '\n",
            "         \"Mastodon server. I haven't really had to do this much but people \"\n",
            "         'talk about it a lot and I was confused about the difference, so:\\n'\n",
            "         '\\n'\n",
            "         '  * A server can **defederate** from another server (this seems to '\n",
            "         'be called '\n",
            "         '[suspend](https://docs.joinmastodon.org/admin/moderation/#suspend-server) '\n",
            "         'in the Mastodon docs). This means that nobody on a server can follow '\n",
            "         'someone from the other server.\\n'\n",
            "         '  * A server can '\n",
            "         '**[limit](https://docs.joinmastodon.org/admin/moderation/#limit-user)** '\n",
            "         '(also known as \"silence\") a user or server. This means that content '\n",
            "         \"from that user is only visible to that user's followers - people \"\n",
            "         'can\\'t discover the user through retweets (aka \"boosts\" on '\n",
            "         'Mastodon).\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"One thing that wasn't obvious to me is that who servers defederate / \"\n",
            "         \"limit is sometimes hidden, so it's hard to suss out what's going on \"\n",
            "         \"if you're considering joining a server, or trying to understand why \"\n",
            "         \"you can't see certain posts.\\n\"\n",
            "         '\\n'\n",
            "         \"### there's no search for posts\\n\"\n",
            "         '\\n'\n",
            "         \"There's no way to search past posts you've read. If I see something \"\n",
            "         'interesting on my timeline and want to find it later, I usually '\n",
            "         \"can't. (Mastodon has a [Elasticsearch-based search \"\n",
            "         'feature](https://docs.joinmastodon.org/admin/optional/elasticsearch/), '\n",
            "         'but it only allows you to search your own posts, your mentions, your '\n",
            "         'favourites, and your bookmarks)\\n'\n",
            "         '\\n'\n",
            "         'These limitations on search are intentional (and a very common '\n",
            "         \"source of arguments) - it's a privacy / safety issue. Here's a \"\n",
            "         '[summary from Tim '\n",
            "         'Bray](https://www.tbray.org/ongoing/When/202x/2022/12/30/Mastodon-Privacy-and-Search) '\n",
            "         'with lots of links.\\n'\n",
            "         '\\n'\n",
            "         'It would be personally convenient for me to be able to search more '\n",
            "         \"easily but I respect folks' safety concerns so I'll leave it at \"\n",
            "         'that.\\n'\n",
            "         '\\n'\n",
            "         'My understanding is that the Mastodon devs are planning to add '\n",
            "         'opt-in search for public posts relatively soon.\\n'\n",
            "         '\\n'\n",
            "         '### other ActivityPub software\\n'\n",
            "         '\\n'\n",
            "         \"We've been talking about Mastodon a lot, but not everyone who I \"\n",
            "         'follow is using Mastodon: Mastodon uses a protocol called '\n",
            "         '[ActivityPub](https://activitypub.rocks/) to distribute messages.\\n'\n",
            "         '\\n'\n",
            "         'Here are some examples of other software I see people talking about, '\n",
            "         'in no particular order:\\n'\n",
            "         '\\n'\n",
            "         '  * [Calckey](https://calckey.org/)\\n'\n",
            "         '  * [Akkoma](https://akkoma.social/)\\n'\n",
            "         '  * [gotosocial](https://gotosocial.org/)\\n'\n",
            "         '  * [Takah\u0113](https://jointakahe.org/)\\n'\n",
            "         '  * [writefreely](https://writefreely.org/)\\n'\n",
            "         '  * [pixelfed](https://pixelfed.org/) (for images)\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I'm probably missing a bunch of important ones.\\n\"\n",
            "         '\\n'\n",
            "         \"### what's the difference between Mastodon and other ActivityPub \"\n",
            "         'software?\\n'\n",
            "         '\\n'\n",
            "         \"This confused me for a while, and I'm still not super clear on how \"\n",
            "         \"ActivityPub works. What I've understood is:\\n\"\n",
            "         '\\n'\n",
            "         '  * ActivityPub is a protocol (you can explore how it works with '\n",
            "         \"blinry's nice [JSON \"\n",
            "         'explorer](https://json.blinry.org/#https://chaos.social/users/blinry))\\n'\n",
            "         '  * Mastodon **servers** communicate with each other (and with other '\n",
            "         'ActivityPub servers) using ActivityPub\\n'\n",
            "         '  * Mastodon **clients** communicate with their server using the '\n",
            "         'Mastodon API, which is its own thing\\n'\n",
            "         \"  * There's also software like \"\n",
            "         '[GoToSocial](https://github.com/superseriousbusiness/gotosocial) '\n",
            "         'that aims to be compatible with the Mastodon API, so that you can '\n",
            "         'use a Mastodon client with it\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '### more mastodon resources\\n'\n",
            "         '\\n'\n",
            "         '  * [Fedi.Tips](https://fedi.tips/) seems to be a great '\n",
            "         'introduction\\n'\n",
            "         '  * I think you can still use '\n",
            "         '[FediFinder](https://fedifinder.glitch.me/) to find folks you '\n",
            "         'followed on Twitter on Mastodon\\n'\n",
            "         \"  * I've been using the [Ivory](https://tapbots.com/ivory/) client \"\n",
            "         'on iOS, but there are lots of great clients. '\n",
            "         '[Elk](https://elk.zone/) is an alternative web client that folks '\n",
            "         'seem to like.\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"I haven't written here about what Mastodon culture is like because \"\n",
            "         'other people have done a much better job of talking about it than '\n",
            "         \"me, but of course it's is the biggest thing that affects your \"\n",
            "         'experience and it was the thing that took me longest to get a handle '\n",
            "         'on. A few links:\\n'\n",
            "         '\\n'\n",
            "         '  * Erin Kissane on [frictions people run into when joining '\n",
            "         'Mastodon](https://erinkissane.com/mastodon-is-easy-and-fun-except-when-it-isnt)\\n'\n",
            "         '  * Kyle Kingsbury wrote some great [moderation guidelines for '\n",
            "         'woof.group](https://blog.woof.group/mods/) (note: woof.group is a '\n",
            "         'LGBTQ+ leather instance, be prepared to see lots of NSFW posts if '\n",
            "         'you visit it)\\n'\n",
            "         '  * Mekka Okereke writes [lots of great posts about issues Black '\n",
            "         'people encounter on '\n",
            "         'Mastodon](https://hachyderm.io/@mekkaokereke/110273797004251326) '\n",
            "         \"(though they're all on Mastodon so it's a little hard to navigate)\\n\"\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         '\\n'\n",
            "         \"### that's all!\\n\"\n",
            "         '\\n'\n",
            "         \"I don't regret setting up a single-user server - even though it's \"\n",
            "         \"inconvenient, it's important to me to have control over my social \"\n",
            "         'media. I think \"have control over my social media\" is more important '\n",
            "         'to me than it is to most other people though, because I use '\n",
            "         'Twitter/Mastodon a lot for work.\\n'\n",
            "         '\\n'\n",
            "         \"I am happy that I didn't _start out_ on a single-user server though \"\n",
            "         '- I think it would have made getting started on Mastodon a lot more '\n",
            "         'difficult.\\n'\n",
            "         '\\n'\n",
            "         \"Mastodon is pretty rough around the edges sometimes but I'm able to \"\n",
            "         'have more interesting conversations about computers there than I am '\n",
            "         \"on Twitter (or Bluesky), so that's where I'm staying for now.\\n\",\n",
            " 'title': 'Notes on using a single-person Mastodon server',\n",
            " 'updated': '2023-08-11T08:13:16+00:00'}\n"
          ]
        }
      ],
      "source": [
        "pprint.pprint(blogposts[0])\n",
        "\n",
        "documents = []\n",
        "for blogpost in blogposts:\n",
        "    metadata = {}\n",
        "    metadata['author'] = blogpost['author']\n",
        "    metadata['url'] = blogpost['link']\n",
        "    metadata['updated_at'] = blogpost['updated']\n",
        "    metadata['title'] = blogpost['title']\n",
        "    document = llama_index.Document(text=blogpost['text'], id_=blogpost['title'], metadata=metadata)\n",
        "    documents.append(document)\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "22637948-dd35-401d-b30e-c7a13825cc07",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create an index from these new documents. Retrieve 20 search results about Nix (a package manager).\n",
        "\n",
        "index = llama_index.VectorStoreIndex.from_documents(documents, service_context=service_context)\n",
        "\n",
        "retriever = index.as_retriever(similarity_top_k = 20)\n",
        "results = retriever.retrieve(\"Nix\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bc456b4b-1818-4bfe-b68e-78adb9796ca4",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.75\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.70\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.68\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.68\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.67\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.67\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.67\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.67\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.66\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.66\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.66\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.66\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.66\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.65\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.65\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.65\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.64\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.64\thttps://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/\t2023-02-28T23:16:17+00:00\n",
            "0.64\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n",
            "0.63\thttps://jvns.ca/blog/2023/03/03/how-do-nix-builds-work-/\t2023-03-03T11:19:25+00:00\n"
          ]
        }
      ],
      "source": [
        "# For all 20 chunks, print their score, url and the date the blogpost was published. \n",
        "\n",
        "for result in results:\n",
        "    print(f\"{result.score:.2f}\\t{result.node.metadata['url']}\\t{result.node.metadata['updated_at']}\")"
      ]
    }
  ],
  "nbformat": 4,
  "nbformat_minor": 5
}