daemon
released this
23 Apr 21:08
Change evaluation methodology (#2)
* Change evaluation methodology
- Make MRR, P@1, and R@3 default
- Add natural language queries and keyword queries
- Add 9 more QD pairs
* Physical sciences topics
* Add temperature questions
* Add more examples
* Implement QA transformer reranker
* Fix dataset typo
* Implement random baseline
* Implement more reranker model types
- Implement question answering reranker
- Implement sequence classification reranker
* Add final dataset examples
* Remove missing IDs from dataset
* Improve cosine similarity matrix provider name
* Add dataset statistics calculation
* Add version to dataset
* Add random MRR calculation
* Fix off-by-one in random MRR computation
* Prepare for release
- Add setuptools script
- Fix circular imports
* Add more detailed README
* Change license to Apache
* Fix classifier name
* Clarify README
Co-authored-by: Nikhil Gupta <[email protected]>
Co-authored-by: Edwin Zhang <[email protected]>