Skip to content

Commit

Permalink
update 2cr report (#1507)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jacklin64
jacklin64 authored Apr 21, 2023
1 parent b02bb8f commit ca5a2be
Showing 1 changed file with 212 additions and 0 deletions.
212 changes: 212 additions & 0 deletions docs/2cr/msmarco-v1-passage.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2885,6 +2885,214 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
</div>
<!-- Tabs content -->

</div></td>
</tr>
<!-- Condition: Aggretriever-DistilBERT: on-the-fly query inference -->
<tr class="accordion-toggle collapsed" id="row27" data-toggle="collapse" data-parent="#row27" href="#collapse27">
<td class="expand-button"></td>
<td style="min-width: 85px">[7]</td>
<td style="min-width: 400px">Aggretriever-DistilBERT: on-the-fly query inference</td>
<td>0.4301</td>
<td>0.6816</td>
<td>0.8023</td>
<td></td>
<td>0.4329</td>
<td>0.6726</td>
<td>0.8351</td>
<td></td>
<td>0.3412</td>
<td>0.9604</td>
</tr>
<tr class="hide-table-padding">
<td></td>
<td colspan="11">
<div id="collapse27" class="collapse in p-3">

<!-- Tabs navs -->
<ul class="nav nav-tabs mb-3" id="row27-tabs" role="tablist">
<li class="nav-item" role="presentation">
<a class="nav-link active" id="row27-tab1-header" data-mdb-toggle="tab" href="#row27-tab1" role="tab" aria-controls="row27-tab1" aria-selected="true" style="text-transform:none">TREC 2019</a>
</li>
<li class="nav-item" role="presentation">
<a class="nav-link" id="row27-tab2-header" data-mdb-toggle="tab" href="#row27-tab2" role="tab" aria-controls="row27-tab2" aria-selected="false" style="text-transform:none">TREC 2020</a>
</li>
<li class="nav-item" role="presentation">
<a class="nav-link" id="row27-tab3-header" data-mdb-toggle="tab" href="#row27-tab3" role="tab" aria-controls="row27-tab3" aria-selected="false" style="text-transform:none">dev</a>
</li>
</ul>
<!-- Tabs navs -->

<!-- Tabs content -->
<div class="tab-content" id="row27-content">
<div class="tab-pane fade show active" id="row27-tab1" role="tabpanel" aria-labelledby="row27-tab1">
Command to generate run on TREC 2019 queries:

<blockquote class="mycode">
<pre><code>python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--index msmarco-passage.aggretriever-distilbert \
--topics dl19-passage \
--encoder castorini/aggretriever-distilbert \
--output run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl19.txt
</code></pre></blockquote>
Evaluation commands:

<blockquote class="mycode">
<pre><code>python -m pyserini.eval.trec_eval -c -l 2 -m map dl19-passage run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl19.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-passage run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl19.txt
python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl19-passage run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl19.txt
</code></pre>
</blockquote>

</div>
<div class="tab-pane fade" id="row27-tab2" role="tabpanel" aria-labelledby="row27-tab2">
Command to generate run on TREC 2020 queries:

<blockquote class="mycode">
<pre><code>python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--index msmarco-passage.aggretriever-distilbert \
--topics dl20 \
--encoder castorini/aggretriever-distilbert \
--output run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl20.txt
</code></pre></blockquote>
Evaluation commands:

<blockquote class="mycode">
<pre><code>python -m pyserini.eval.trec_eval -c -l 2 -m map dl20-passage run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-passage run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl20.txt
python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl20-passage run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dl20.txt
</code></pre>
</blockquote>

</div>
<div class="tab-pane fade" id="row27-tab3" role="tabpanel" aria-labelledby="row27-tab3">
Command to generate run on dev queries:

<blockquote class="mycode">
<pre><code>python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--index msmarco-passage.aggretriever-distilbert \
--topics msmarco-passage-dev-subset \
--encoder castorini/aggretriever-distilbert \
--output run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dev.txt
</code></pre></blockquote>
Evaluation commands:

<blockquote class="mycode">
<pre><code>python -m pyserini.eval.trec_eval -c -M 10 -m recip_rank msmarco-passage-dev-subset run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-passage-dev-subset run.msmarco-v1-passage.Aggretriever-Distilbert-otf.dev.txt
</code></pre>
</blockquote>

</div>
</div>
<!-- Tabs content -->

</div></td>
</tr>
<!-- Condition: Aggretriever-coCondenser: on-the-fly query inference -->
<tr class="accordion-toggle collapsed" id="row28" data-toggle="collapse" data-parent="#row28" href="#collapse28">
<td class="expand-button"></td>
<td style="min-width: 85px">[7]</td>
<td style="min-width: 400px">Aggretriever-coCondenser: on-the-fly query inference</td>
<td>0.4350</td>
<td>0.6837</td>
<td>0.8078</td>
<td></td>
<td>0.4710</td>
<td>0.6972</td>
<td>0.8555</td>
<td></td>
<td>0.3619</td>
<td>0.9735</td>
</tr>
<tr class="hide-table-padding">
<td></td>
<td colspan="11">
<div id="collapse28" class="collapse in p-3">

<!-- Tabs navs -->
<ul class="nav nav-tabs mb-3" id="row28-tabs" role="tablist">
<li class="nav-item" role="presentation">
<a class="nav-link active" id="row28-tab1-header" data-mdb-toggle="tab" href="#row28-tab1" role="tab" aria-controls="row28-tab1" aria-selected="true" style="text-transform:none">TREC 2019</a>
</li>
<li class="nav-item" role="presentation">
<a class="nav-link" id="row28-tab2-header" data-mdb-toggle="tab" href="#row28-tab2" role="tab" aria-controls="row28-tab2" aria-selected="false" style="text-transform:none">TREC 2020</a>
</li>
<li class="nav-item" role="presentation">
<a class="nav-link" id="row28-tab3-header" data-mdb-toggle="tab" href="#row28-tab3" role="tab" aria-controls="row28-tab3" aria-selected="false" style="text-transform:none">dev</a>
</li>
</ul>
<!-- Tabs navs -->

<!-- Tabs content -->
<div class="tab-content" id="row28-content">
<div class="tab-pane fade show active" id="row28-tab1" role="tabpanel" aria-labelledby="row28-tab1">
Command to generate run on TREC 2019 queries:

<blockquote class="mycode">
<pre><code>python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--index msmarco-passage.aggretriever-cocondenser \
--topics dl19-passage \
--encoder castorini/aggretriever-cocondenser \
--output run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl19.txt
</code></pre></blockquote>
Evaluation commands:

<blockquote class="mycode">
<pre><code>python -m pyserini.eval.trec_eval -c -l 2 -m map dl19-passage run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl19.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl19-passage run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl19.txt
python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl19-passage run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl19.txt
</code></pre>
</blockquote>

</div>
<div class="tab-pane fade" id="row28-tab2" role="tabpanel" aria-labelledby="row28-tab2">
Command to generate run on TREC 2020 queries:

<blockquote class="mycode">
<pre><code>python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--index msmarco-passage.aggretriever-cocondenser \
--topics dl20 \
--encoder castorini/aggretriever-cocondenser \
--output run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl20.txt
</code></pre></blockquote>
Evaluation commands:

<blockquote class="mycode">
<pre><code>python -m pyserini.eval.trec_eval -c -l 2 -m map dl20-passage run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl20.txt
python -m pyserini.eval.trec_eval -c -m ndcg_cut.10 dl20-passage run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl20.txt
python -m pyserini.eval.trec_eval -c -l 2 -m recall.1000 dl20-passage run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dl20.txt
</code></pre>
</blockquote>

</div>
<div class="tab-pane fade" id="row28-tab3" role="tabpanel" aria-labelledby="row28-tab3">
Command to generate run on dev queries:

<blockquote class="mycode">
<pre><code>python -m pyserini.search.faiss \
--threads 16 --batch-size 512 \
--index msmarco-passage.aggretriever-cocondenser \
--topics msmarco-passage-dev-subset \
--encoder castorini/aggretriever-cocondenser \
--output run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dev.txt
</code></pre></blockquote>
Evaluation commands:

<blockquote class="mycode">
<pre><code>python -m pyserini.eval.trec_eval -c -M 10 -m recip_rank msmarco-passage-dev-subset run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dev.txt
python -m pyserini.eval.trec_eval -c -m recall.1000 msmarco-passage-dev-subset run.msmarco-v1-passage.Aggretriever-coCondenser-otf.dev.txt
</code></pre>
</blockquote>

</div>
</div>
<!-- Tabs content -->

</div></td>
</tr>

Expand Down Expand Up @@ -2920,6 +3128,10 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<a href="https://arxiv.org/abs/2302.06587">SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes.</a>
<i>arXiv:2302.06587</i>, Feburary 2023.</p></li>

<li><p>[7] Sheng-Chieh Lin, Minghan Li and Jimmy Lin.
<a href="https://arxiv.org/abs/2208.00511">Aggretriever: A Simple Approach to Aggregate Textual Representation for Robust Dense Passage Retrieval.</a>
<i>arXiv:2208.00511</i>, July 2022.</p></li>


</ul>

Expand Down

0 comments on commit ca5a2be

Please sign in to comment.