-
Notifications
You must be signed in to change notification settings - Fork 8
/
CITATION.cff
62 lines (61 loc) · 2.28 KB
/
CITATION.cff
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
cff-version: 1.2.0
type: software
title: TIRA
repository-code: 'https://github.com/tira-io/tira'
url: 'https://tira.io/'
license: MIT
preferred-citation:
type: conference-paper
title: Continuous Integration for Reproducible Shared Tasks with TIRA.io
collection-title: Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023)
location: Dublin, Irland
year: 2023
month: 4
pages: 236--241
conference:
name: Lecture Notes in Computer Science
city: Berlin, Heidelberg, New York
publisher:
name: Springer
doi: doi.org/10.1007/978-3-031-28241-6_20
url: https://doi.org/10.1007/978-3-031-28241-6_20
authors:
- given-names: Maik
family-names: Fr{\"o}be
- given-names: Matti
family-names: Wiegmann
- given-names: Nikolay
family-names: Kolyada
- given-names: Bastian
family-names: Grahm
- given-names: Theresa
family-names: Elstner
- given-names: Frank
family-names: Loebe
- given-names: Matthias
family-names: Hagen
- given-names: Benno
family-names: Stein
- given-names: Martin
family-names: Potthast
abstract: >-
A major obstacle to the long-term impact of most shared
tasks is their lack of reproducibility. Often only the
test collections and the papers of the organizers and
participants are published. Third parties who want to
independently evaluate the state of the art for a task on
other data must re-implement the participants’ software.
The tools developed to collect software from participants
in shared tasks only partially verify its reliability at
the time of submission, much less long-term, and do not
enable third parties to reuse it later. We have overhauled
the TIRA Integrated Research Architecture to address all
of these issues. The new version simplifies task setup for
organizers and software submission for participants,
scales from a local computer to the cloud, supports
on-demand resource allocation up to parallel CPU and GPU
processing, and enables export for local reproduction with
just a few lines of code. This is achieved by implementing
the TIRA protocol with an industry-standard continuous
integration and deployment (CI/CD) pipeline using Git,
Docker, and Kubernetes.