reproducibility-experiments/re-rank-reproducibility.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "08ce106f-b0ee-476b-9a46-710cd09f954f",
   "metadata": {},
   "source": [
    "# Reproducing Re-Rankers\n",
    "\n",
    "This notebook shows how post-hoc experiments of the IR Experiment Platform can be conducted.\n",
    "\n",
    "To start the notebook, please clone the archived shared task repository:\n",
    "\n",
    "```\n",
    "git@github.com:tira-io/ir-experiment-platform-benchmarks.git\n",
    "```\n",
    "\n",
    "Inside the cloned repository, you can start the Jupyter notebook which automatically installs a minimal virtual environment using:\n",
    "```\n",
    "make jupyterlab\n",
    "```\n",
    "\n",
    "The notebook covers how to run re-rankers submitted to TIRA in reproducibility/replicability experiments on the same or new data.\n",
    "\n",
    "For each of the softwares submitted to TIRA, the `tira` integration to PyTerrier loads the Docker Image submitted to TIRA to execute it in PyTerrier pipelines (i.e., a first execution could take sligthly longer).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c36da51-6d12-444a-8385-a37f42af7781",
   "metadata": {},
   "source": [
    "## Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ce1afbb2-01e9-4ed3-af69-fb1848d634ac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 0)\n",
    "\n",
    "from tira.local_client import Client\n",
    "tira = Client()\n",
    "\n",
    "import pyterrier as pt\n",
    "if not pt.started():\n",
    "    pt.init()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d1134a-2c01-4549-9b3b-74fe987ab304",
   "metadata": {},
   "source": [
    "### Initialize A Re-Ranker\n",
    "\n",
    "We create a pyterrier re-ranker called `advanced_pipeline` that re-ranks BM25 with an re-ranker submitted to a shared task in TIRA.\n",
    "The reranker is identified by the name `ir-benchmarks/tira-ir-starter/SBERT multi-qa-MiniLM-L6-dot-v1 (tira-ir-starter-beir)`, i.e., a software `SBERT multi-qa-MiniLM-L6-dot-v1 (tira-ir-starter-beir)` submitted to `ir-benchmarks` by the team `tira-ir-starter` (that hosts baselines).\n",
    "This software consists a single stage that re-ranks with a dense retrieval approach implemented in BEIR.\n",
    "\n",
    "With this API, any re-rank approach submitted in TIRA can be executed and re-executed, e.g., on new data.\n",
    "\n",
    "We can run the re-ranker on any dataset integrated in `ir_dataset` or any dataframe.\n",
    "Here, we use a small artificial reranking dataset `data_to_rerank` to show the overall functionality with a fast example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c7c75b21-3dca-4251-a703-acff1feb6e67",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>docno</th>\n",
       "      <th>body</th>\n",
       "      <th>qid</th>\n",
       "      <th>query</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d1</td>\n",
       "      <td>this is the first document of many documents</td>\n",
       "      <td>1</td>\n",
       "      <td>first document</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2</td>\n",
       "      <td>this is another document</td>\n",
       "      <td>1</td>\n",
       "      <td>first document</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d3</td>\n",
       "      <td>the topic of this document is unknown</td>\n",
       "      <td>1</td>\n",
       "      <td>first document</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  docno                                          body qid           query\n",
       "0  d1    this is the first document of many documents  1   first document\n",
       "1  d2    this is another document                      1   first document\n",
       "2  d3    the topic of this document is unknown         1   first document"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_to_rerank = pd.DataFrame([\n",
    "        [\"d1\", \"this is the first document of many documents\", \"1\", \"first document\"],\n",
    "        [\"d2\", \"this is another document\", \"1\", \"first document\"],\n",
    "        [\"d3\", \"the topic of this document is unknown\", \"1\", \"first document\"]\n",
    "    ], columns=[\"docno\", \"body\", \"qid\", \"query\"])\n",
    "\n",
    "data_to_rerank"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d523fe7-adc0-4dbd-9574-cc33e2900b4e",
   "metadata": {},
   "source": [
    "First, we re-rank this via BM25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8e19fe7a-63fe-4b1f-8bc2-40cea6cedc26",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>docno</th>\n",
       "      <th>body</th>\n",
       "      <th>qid</th>\n",
       "      <th>rank</th>\n",
       "      <th>score</th>\n",
       "      <th>query</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d1</td>\n",
       "      <td>this is the first document of many documents</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5.560003e-01</td>\n",
       "      <td>first document</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2</td>\n",
       "      <td>this is another document</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>-3.085859e-10</td>\n",
       "      <td>first document</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d3</td>\n",
       "      <td>the topic of this document is unknown</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5.681316e-02</td>\n",
       "      <td>first document</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  docno                                          body qid  rank         score  \\\n",
       "0  d1    this is the first document of many documents  1   0     5.560003e-01   \n",
       "1  d2    this is another document                      1   2    -3.085859e-10   \n",
       "2  d3    the topic of this document is unknown         1   1     5.681316e-02   \n",
       "\n",
       "            query  \n",
       "0  first document  \n",
       "1  first document  \n",
       "2  first document  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bm25 = pt.text.scorer(wmmodel='bm25')\n",
    "\n",
    "bm25(data_to_rerank)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05943808-1fc0-4098-9822-1de3b4db4867",
   "metadata": {},
   "source": [
    "Next, we use the re-ranker `ir-benchmarks/tira-ir-starter/SBERT multi-qa-MiniLM-L6-dot-v1 (tira-ir-starter-beir)` as second stage re-ranker, after BM25.\n",
    "\n",
    "This shows that re-rankers in TIRA are interoperable with other re-rankers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3666ba24-21ce-43aa-a9f1-17bfc4502a7b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>docno</th>\n",
       "      <th>body</th>\n",
       "      <th>qid</th>\n",
       "      <th>query</th>\n",
       "      <th>q0</th>\n",
       "      <th>rank</th>\n",
       "      <th>score</th>\n",
       "      <th>system</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d1</td>\n",
       "      <td>this is the first document of many documents</td>\n",
       "      <td>1</td>\n",
       "      <td>first document</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>46.084885</td>\n",
       "      <td>multi-qa-MiniLM-L6-dot-v1-dot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2</td>\n",
       "      <td>this is another document</td>\n",
       "      <td>1</td>\n",
       "      <td>first document</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>40.802025</td>\n",
       "      <td>multi-qa-MiniLM-L6-dot-v1-dot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d3</td>\n",
       "      <td>the topic of this document is unknown</td>\n",
       "      <td>1</td>\n",
       "      <td>first document</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>37.294750</td>\n",
       "      <td>multi-qa-MiniLM-L6-dot-v1-dot</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  docno                                          body qid           query  q0  \\\n",
       "0  d1    this is the first document of many documents  1   first document  0    \n",
       "1  d2    this is another document                      1   first document  0    \n",
       "2  d3    the topic of this document is unknown         1   first document  0    \n",
       "\n",
       "   rank      score                         system  \n",
       "0  1     46.084885  multi-qa-MiniLM-L6-dot-v1-dot  \n",
       "1  2     40.802025  multi-qa-MiniLM-L6-dot-v1-dot  \n",
       "2  3     37.294750  multi-qa-MiniLM-L6-dot-v1-dot  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "advanced_pipeline = bm25 >> tira.pt.reranker('ir-benchmarks/tira-ir-starter/SBERT multi-qa-MiniLM-L6-dot-v1 (tira-ir-starter-beir)')\n",
    "\n",
    "advanced_pipeline(data_to_rerank)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}