This repository exemplifies how archived shared task repositories of the IR Experiment Platform look like. Here, we have archived the Retrieval Benchmarks in the IR Experiment Platform.
The archived shared task repositories allow post-hoc experiments, and we provide several tutorials with examples in Jupyter Notebooks.
To start the jupyter notebook, please clone the archived shared task repository:
git clone [email protected]:tira-io/ir-experiment-platform-benchmarks.git
Inside the cloned repository, you can start the Jupyter notebook which automatically installs a minimal virtual environment using:
make jupyterlab
The installation of the environment is simplified with a virtual environment and executing make jupyterlab
installs the virtual environment (if not already done) and starts the jupyter notebook ready to run all parts of the tutorial.
For each of the softwares submitted to TIRA, the tira
integration to PyTerrier loads the Docker Image submitted to TIRA to execute it in PyTerrier pipelines (i.e., a first execution could take sligthly longer).
The following tutorial notebooks are available:
- Tutorial.ipynb: A general purpose tutorial showing the first steps.
- full-rank-retriever-tutorial.ipynb: showcases how full-rankers can be reproduced/replicated.
- re-rank-tutorial.ipynb: showcases how re-rankers can be reproduced/replicated.
- interoparability-tutorial.ipynb: showcases how full-rankers and re-rankers submitted in TIRA can be combined in new ways in post-hoc experiments.
Comparing the leaderboards accross different tasks is quite interesting (we have a large scale evaluation on that in the paper), e.g., compare MS MARCO DL 2019 with Antique or Args.me: On MS MARCO, all kinds of deep learning models are at the top, which totally reverses for other corpora, e.g., Args.me or Antique.
The current leaderboards can be viewed in tira.io:
- Antique
- Args.me 2020 Task 1
- Args.me 2021 Task 1
- Cranfield
- TREC COVID
- TREC Deep Learning 2019 (passage)
- TREC Deep Learning 2020 (passage)
- TREC Genomics 2004
- TREC Genomics 2005
- TREC 7
- TREC 8
- Robust04
- TREC Web Track 2002 (gov)
- TREC Web Track 2003 (gov)
- TREC Web Track 2004 (gov)
- TREC Web Track 2009 (ClueWeb09)
- TREC Web Track 2010 (ClueWeb09)
- TREC Web Track 2011 (ClueWeb09)
- TREC Web Track 2012 (ClueWeb09)
- TREC Web Track 2013 (ClueWeb12)
- TREC Web Track 2014 (ClueWeb12)
- Touché 2020 Task 2 (ClueWeb12)
- Touché 2021 Task 2 (ClueWeb12)
- Touché 2023 Task 2 (ClueWeb22) (Task is still ongoing, so the leaderboard is not yet public)
- TREC Terabyte 2004 (gov2)
- TREC Terabyte 2005 (gov2)
- TREC Terabyte 2006 (gov2)
- NFCorpus
- Vaswani
- TREC Core 2018 (wapo)
- TREC Precision Medicine 2017
- TREC Precision Medicine 2018