Add metrics calculations to the inference pipeline #23
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #23 +/- ##
==========================================
+ Coverage 95.83% 96.32% +0.49%
==========================================
Files 3 3
Lines 120 136 +16
==========================================
+ Hits 115 131 +16
Misses 5 5 ☔ View full report in Codecov by Sentry. |
tests/test_main.py
Outdated
| labels = [item["output"] for item in items] | ||
|
|
||
| bleu, meteor = evaluate_documentation(labels, labels) | ||
| assert bleu >= 0 and bleu <= 1, "BLEU score should be between 0 and 1" |
There was a problem hiding this comment.
Are bleu, meteor==1 when label==prediction?
It would actually be a bit clearer if you hard code some examples and assert specific values. Eg. all tokens match, extra token in the prediction, missing token in the prediction, etc.
There was a problem hiding this comment.
Added more tests cases, and code works as expected.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #23 +/- ##
==========================================
+ Coverage 96.15% 96.57% +0.42%
==========================================
Files 3 3
Lines 130 146 +16
==========================================
+ Hits 125 141 +16
Misses 5 5 ☔ View full report in Codecov by Sentry. |
|
Closing the Pull Request for evaluation metrics |
Change Description
Adding changes to add metrics to the inference pipeline in main.py. Added unit test case in test_main.py
closes #2
Solution Description
Added BLEU, METEOR evaluation metrics to the inference pipeline.
BLEU score calculation
https://www.baeldung.com/cs/nlp-bleu-score#:~:text=BLEU%20(Bilingual%20Evaluation%20Understudy)%20is,%2Danswering%20systems%2C%20and%20chatbots.
METEOR score calculation
https://huggingface.co/spaces/evaluate-metric/meteor
Code Quality
Project-Specific Pull Request Checklists