forked from castorini/anserini
-
Notifications
You must be signed in to change notification settings - Fork 0
/
msmarco-passage-doc2query.template
93 lines (60 loc) · 4.27 KB
/
msmarco-passage-doc2query.template
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# Anserini Regressions: MS MARCO Passage Ranking
**Models**: BM25 with (vanilla) doc2query expansions
This page documents regression experiments on the [MS MARCO passage ranking task](https://github.com/microsoft/MSMARCO-Passage-Ranking) with BM25 on (vanilla) doc2query (also called doc2query-base) expansions, as proposed in the following paper:
> Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. [Document Expansion by Query Prediction.](https://arxiv.org/abs/1904.08375) arXiv:1904.08375, 2019.
These experiments are integrated into Anserini's regression testing framework.
For more complete instructions on how to run end-to-end experiments, refer to [this page](experiments-doc2query.md).
The exact configurations for these regressions are stored in [this YAML file](${yaml}).
Note that this page is automatically generated from [this template](${template}) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.
From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:
```
python src/main/python/run_regression.py --index --verify --search --regression ${test_name}
```
## Indexing
Typical indexing command:
```
${index_cmds}
```
The directory `/path/to/msmarco-passage-doc2query` should be a directory containing `jsonl` files containing the expanded passage collection.
[This page](experiments-doc2query.md) explains how to perform this data preparation.
For additional details, see explanation of [common indexing options](common-indexing-options.md).
## Retrieval
Topics and qrels are stored in [`src/main/resources/topics-and-qrels/`](../src/main/resources/topics-and-qrels/).
The regression experiments here evaluate on the 6980 dev set questions; see [this page](experiments-msmarco-passage.md) for more details.
After indexing has completed, you should be able to perform retrieval as follows:
```
${ranking_cmds}
```
Evaluation can be performed using `trec_eval`:
```
${eval_cmds}
```
## Effectiveness
With the above commands, you should be able to reproduce the following results:
${effectiveness}
Explanation of settings:
+ The setting "default" refers the default BM25 settings of `k1=0.9`, `b=0.4`.
+ The setting "tuned" refers to `k1=0.82`, `b=0.68`, tuned on _on the original passages_, as described in [this page](experiments-msmarco-passage.md).
## Additional Implementation Details
Note that prior to December 2021, runs generated with `SearchCollection` in the TREC format and then converted into the MS MARCO format give slightly different results from runs generated by `SearchMsmarco` directly in the MS MARCO format, due to tie-breaking effects.
This was fixed with [#1458](https://github.com/castorini/anserini/issues/1458), which also introduced (intra-configuration) multi-threading.
As a result, `SearchMsmarco` has been deprecated and replaced by `SearchCollection`; both have been verified to generate _identical_ output.
The commands below have been retained for historical reasons only.
The following command generates with `SearchMsmarco` the run denoted "BM25 (tuned)" above (`k1=0.82`, `b=0.68`):
```bash
$ sh target/appassembler/bin/SearchMsmarco -hits 1000 -threads 8 \
-index indexes/lucene-index.msmarco-passage-doc2query.pos+docvectors+raw \
-queries collections/msmarco-passage/queries.dev.small.tsv \
-k1 0.82 -b 0.68 \
-output runs/run.msmarco-passage-doc2query
$ python tools/scripts/msmarco/msmarco_passage_eval.py \
collections/msmarco-passage/qrels.dev.small.tsv runs/run.msmarco-passage-doc2query
#####################
MRR @10: 0.2213412471005586
QueriesRanked: 6980
#####################
```
Note that this run does _not_ correspond to the scores reported in the paper that introduced doc2query:
> Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. [Document Expansion by Query Prediction.](https://arxiv.org/abs/1904.08375) arXiv:1904.08375, 2019.
The scores reported in the above paper refer to entry "BM25 (Anserini) + doc2query" dated 2019/04/10 on the [MS MARCO Passage Ranking Leaderboard](https://microsoft.github.io/msmarco/).
The paper/leaderboard run reports 0.215 MRR@10, which is slightly lower than the "BM25 (Tuned)" regression run above, due to an earlier version of Lucene (7.6) and use of default BM25 parameters.