Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial implementation of repro matrix for MS MARCO v1 passage #1127

Merged
merged 6 commits into from
Apr 27, 2022

Conversation

lintool
Copy link
Member

@lintool lintool commented Apr 26, 2022

The goal is a repo that will automatically generate the results table in the SIGIR 2022 resource paper. This is my initial implementation.

After running python scripts/run_all_msmarco_v1_passage.py, the following is automatically generated:


                                                 TREC 2019                TREC 2020            MS MARCO dev
                                             MAP nDCG@10    R@1K       MAP nDCG@10    R@1K    MRR@10    R@1K
                                          ----------------------    ----------------------    --------------
BM25 (k1=0.82, b=0.68)                    0.2903  0.4973  0.7450    0.2876  0.4876  0.8031    0.1875  0.8573
BM25+RM3 (k1=0.82, b=0.68)                0.3377  0.5231  0.7792    0.3056  0.4808  0.8286    0.1668  0.8687
BM25 w/ doc2query-T5 (k1=2.18, b=0.86)    0.4046  0.6336  0.8134    0.4171  0.6265  0.8393    0.2816  0.9506

BM25 (k1=0.9, b=0.4)                      0.3013  0.5058  0.7501    0.2856  0.4796  0.7863    0.1840  0.8526
BM25+RM3 (k1=0.9, b=0.4)                  0.3390  0.5180  0.7998    0.3019  0.4821  0.8217    0.1564  0.8606
BM25 w/ doc2query-T5 (k1=0.9, b=0.4)      0.4034  0.6417  0.8310    0.4074  0.6187  0.8452    0.2723  0.9470

uniCOIL (w/ doc2query-T5) - pre-encoded   0.4612  0.7024  0.8292    0.4430  0.6745  0.8430    0.3516  0.9582
uniCOIL (w/ doc2query-T5) - otf           0.4617  0.7027  0.8291    0.4429  0.6745  0.8433    0.3509  0.9581

For reference, here is the results table in the paper:

Screen Shot 2022-04-26 at 7 44 22 PM

Copy link
Member

@ronakice ronakice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@lintool
Copy link
Member Author

lintool commented Apr 27, 2022

We can even incorporate dense retrieval:

                                                 TREC 2019                TREC 2020            MS MARCO dev
                                             MAP nDCG@10    R@1K       MAP nDCG@10    R@1K    MRR@10    R@1K
                                          ----------------------    ----------------------    --------------
BM25 (k1=0.82, b=0.68)                    0.2903  0.4973  0.7450    0.2876  0.4876  0.8031    0.1875  0.8573
BM25+RM3 (k1=0.82, b=0.68)                0.3377  0.5231  0.7792    0.3056  0.4808  0.8286    0.1668  0.8687
BM25 w/ doc2query-T5 (k1=2.18, b=0.86)    0.4046  0.6336  0.8134    0.4171  0.6265  0.8393    0.2816  0.9506

BM25 (k1=0.9, b=0.4)                      0.3013  0.5058  0.7501    0.2856  0.4796  0.7863    0.1840  0.8526
BM25+RM3 (k1=0.9, b=0.4)                  0.3390  0.5180  0.7998    0.3019  0.4821  0.8217    0.1564  0.8606
BM25 w/ doc2query-T5 (k1=0.9, b=0.4)      0.4034  0.6417  0.8310    0.4074  0.6187  0.8452    0.2723  0.9470

uniCOIL (w/ doc2query-T5) - pre-encoded   0.4612  0.7024  0.8292    0.4430  0.6745  0.8430    0.3516  0.9582
uniCOIL (w/ doc2query-T5) - otf           0.4617  0.7027  0.8291    0.4429  0.6745  0.8433    0.3509  0.9581

TCT_ColBERT-V2-HN+ - pre-encoded          0.0000  0.0000  0.0000    0.0000  0.0000  0.0000    0.3584  0.9695
TCT_ColBERT-V2-HN+ - otf                  0.4469  0.7204  0.8261    0.4754  0.6882  0.8429    0.3584  0.9695

@lintool lintool merged commit 244828f into master Apr 27, 2022
@lintool lintool deleted the matrix branch April 27, 2022 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants