Fix splade-pp-sd-onnx regression tests (#2253)

fix inconsistent scores and commands
castorini · Nov 9, 2023 · 75c553f · 75c553f
1 parent d152e5a
commit 75c553f
Show file tree

Hide file tree

Showing 2 changed files with 13 additions and 13 deletions.
diff --git a/docs/regressions/regressions-msmarco-passage-splade-pp-sd-onnx.md b/docs/regressions/regressions-msmarco-passage-splade-pp-sd-onnx.md
@@ -75,19 +75,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 ```bash
 target/appassembler/bin/SearchCollection \
   -index indexes/lucene-index.msmarco-passage-splade-pp-sd/ \
-  -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.splade-pp-sd.tsv.gz \
+  -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.txt \
   -topicreader TsvInt \
-  -output runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.splade-pp-sd.txt \
-  -impact -pretokenized &
+  -output runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.txt \
+  -impact -pretokenized -encoder SpladePlusPlusSelfDistil &
 ```
 
 Evaluation can be performed using `trec_eval`:
 
 ```bash
-tools/eval/trec_eval.9.0.4/trec_eval -c -m map tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.splade-pp-sd.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -M 10 -m recip_rank tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.splade-pp-sd.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.splade-pp-sd.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.splade-pp-sd.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m map tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 10 -m recip_rank tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 tools/topics-and-qrels/qrels.msmarco-passage.dev-subset.txt runs/run.msmarco-passage-splade-pp-sd.splade-pp-sd.topics.msmarco-passage.dev-subset.txt
 ```
 
 ## Effectiveness
@@ -96,9 +96,9 @@ With the above commands, you should be able to reproduce the following results:
 
 | **AP@1000**                                                                                                  | **SPLADE++ CoCondenser-SelfDistil**|
 |:-------------------------------------------------------------------------------------------------------------|-----------|
-| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)                                | 0.3837    |
+| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)                                | 0.3835    |
 | **RR@10**                                                                                                    | **SPLADE++ CoCondenser-SelfDistil**|
-| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)                                | 0.3778    |
+| [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)                                | 0.3776    |
 | **R@100**                                                                                                    | **SPLADE++ CoCondenser-SelfDistil**|
 | [MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)                                | 0.9118    |
 | **R@1000**                                                                                                   | **SPLADE++ CoCondenser-SelfDistil**|

diff --git a/src/main/resources/regression/msmarco-passage-splade-pp-sd-onnx.yaml b/src/main/resources/regression/msmarco-passage-splade-pp-sd-onnx.yaml
@@ -49,18 +49,18 @@ topic_reader: TsvInt
 topics:
   - name: "[MS MARCO Passage: Dev](https://github.com/microsoft/MSMARCO-Passage-Ranking)"
     id: dev
-    path: topics.msmarco-passage.dev-subset.splade-pp-sd.tsv.gz
+    path: topics.msmarco-passage.dev-subset.txt
     qrel: qrels.msmarco-passage.dev-subset.txt
 
 models:
   - name: splade-pp-sd
     display: SPLADE++ CoCondenser-SelfDistil
-    params: -impact -pretokenized
+    params: -impact -pretokenized -encoder SpladePlusPlusSelfDistil
     results:
       AP@1000:
-        - 0.3837
+        - 0.3835
       RR@10:
-        - 0.3778
+        - 0.3776
       R@100:
         - 0.9118
       R@1000: