RealTime QA is a dynamic question answering (QA) platform that inquires about the present. We announce ~30 questions and evaluate real-time baseline systems (e.g., GPT-3 and T5) on a regular basis (weekly in this version).
Follow us on Twitter. We tweet interesting examples from our GPT-3 baselines every week!
This repository provides our past data, as well as the latest, ongoing questions.
The backnumber questions preceded our real-time baselines that started from June 17, 2022. See our paper for more detail. Our baseline retrieval results (Google custom search and DPR) are also provided (e.g., June 17, 2022 Google custom search). The past questions have all six baseline results under the multiple-choice and original settings (e.g., GPT-3 + Google Custom Search Generation Results).
The submission window closes when the next set of questions is announced (3 am GMT on every Saturday). Submit via this Google form. The indexes for multiple-choice answer should start with 0. See examples.
See our script that we use for the six baseline models in the paper. We run them every week. The results are updated every week on our website.
@inproceedings{
kasai2023realtime,
title={RealTime {QA}: What's the Answer Right Now?},
author={Jungo Kasai and Keisuke Sakaguchi and yoichi takahashi and Ronan Le Bras and Akari Asai and Xinyan Velocity Yu and Dragomir Radev and Noah A. Smith and Yejin Choi and Kentaro Inui},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023},
url={https://openreview.net/forum?id=HfKOIPCvsv}
}
