Skip to content

Add completions_logprob endpoint and extend BYOB scorers/datasets for Sovereign benchmarks - #1

Closed
kanishks-23 wants to merge 3 commits into
mainfrom
kanishks/sovereign_benchmarks
Closed

Add completions_logprob endpoint and extend BYOB scorers/datasets for Sovereign benchmarks#1
kanishks-23 wants to merge 3 commits into
mainfrom
kanishks/sovereign_benchmarks

Conversation

@kanishks-23

@kanishks-23 kanishks-23 commented Apr 29, 2026

Copy link
Copy Markdown
Owner

BYOB support for logprob-based multiple-choice evaluation and extends dataset loading/scoring utilities needed by the Sovereign benchmark suite.

  • Added completions_logprob as a supported endpoint type
    • Pydantic validation now accepts target.api_endpoint.type: completions_logprob
    • CLI --model_type completions_logprob is allowed
    • Endpoint health checks treat completions_logprob like a /v1/completions endpoint
  • Added BYOB logprob evaluation flow
    • Uses /v1/completions with max_tokens=0, echo=true, and logprobs=1
    • Scores candidate continuations using returned token logprobs
    • Supports multiple-choice ranking via one request per candidate answer
    • Supports nested choice fields such as choices.text
  • Added/updated BYOB scorers
    • multiple_choice_acc: returns acc, acc_norm, and acc_greedy
    • mcq_letter_extract: supports A-J options and handles empty/None responses safely
    • Added task-oriented scorers for GSM8K-style numeric answers, yes/no tasks, chrF, and ROUGE
  • Extended Hugging Face dataset URI support.
    • Parses extra query params beyond split
    • Supports trust_remote_code=true
    • Supports row filtering via filter_field/filter_value, including multiple filters with suffixes
    • This allows BYOB benchmarks to consume datasets where language is stored as a row field instead of a HF config

@kanishks-23
kanishks-23 force-pushed the kanishks/sovereign_benchmarks branch 2 times, most recently from d5f1ed8 to 4a0cee3 Compare April 29, 2026 14:31
@kanishks-23 kanishks-23 changed the title Add custom BYOB logprob scoring support Add completions_logprob endpoint and extend BYOB scorers/datasets for Sovereign benchmarks Apr 29, 2026
@kanishks-23
kanishks-23 force-pushed the kanishks/sovereign_benchmarks branch from af194da to 3ac7a3e Compare April 30, 2026 10:08
wprazuch
wprazuch approved these changes Apr 30, 2026
View reviewed changes
Comment thread docs/libraries/nemo-evaluator/extending/byob/scorers.md Outdated
Comment thread packages/nemo-evaluator/src/nemo_evaluator/contrib/byob/decorators.py Outdated
wprazuch
wprazuch reviewed Apr 30, 2026
View reviewed changes
Comment thread packages/nemo-evaluator/src/nemo_evaluator/contrib/byob/compiler.py Outdated
Comment thread packages/nemo-evaluator/src/nemo_evaluator/contrib/byob/compiler.py Outdated
Comment thread packages/nemo-evaluator/src/nemo_evaluator/contrib/byob/compiler.py Outdated
@kanishks-23
kanishks-23 force-pushed the kanishks/sovereign_benchmarks branch from b241165 to ca07cb8 Compare May 3, 2026 14:28
@wprazuch wprazuch mentioned this pull request May 4, 2026
Draft
7 tasks
@kanishks-23
kanishks-23 force-pushed the kanishks/sovereign_benchmarks branch from ca07cb8 to cdaa77a Compare May 4, 2026 15:15
@github-actions github-actions Bot added documentation Improvements or additions to documentation nemo-evaluator tests labels May 4, 2026
@kanishks-23
kanishks-23 force-pushed the kanishks/sovereign_benchmarks branch 2 times, most recently from 5ae25d7 to bb06b86 Compare May 5, 2026 07:57
@kanishks-23
feat(byob): add completions_logprob endpoint and extend scorers/datasets
45e9484 
Add support for logprob-based multiple-choice evaluation within the BYOB
framework, plus several new scorers and dataset-loading enhancements.

Endpoint and runner:
- Add completions_logprob as a supported endpoint type with Pydantic
  validation, CLI support, and health-check handling
- Logprob scoring uses /v1/completions with max_tokens=0, echo=true,
  logprobs=1 to score candidate answers
- New call_model_loglikelihood helper in runner.py

Scorers:
- multiple_choice_acc: logprob-based MCQ scoring
- mcq_letter_extract: parse first A-D letter from free-form responses
- gsm8k_answer: GSM8K-style numeric answer extraction
- boolean_yesno: yes/no task scorer
- chrf / chrF++ (sacreBLEU-style character n-gram F-score)
- Extended ROUGE support

Datasets:
- HuggingFace loader parses additional query parameters
- trust_remote_code=true enabled where required

Signed-off-by: kanishks <kanishks@nvidia.com>
@kanishks-23
kanishks-23 force-pushed the kanishks/sovereign_benchmarks branch from bb06b86 to 45e9484 Compare May 5, 2026 08:57
kanishks-23 added 2 commits May 5, 2026 21:18
@kanishks-23
fix(byob): log warning when choices_field falls back to static choices
6cdca7e 
  When @benchmark(choices_field=...) is set but the row doesn't expose
  that field, eval_logic silently falls back to @benchmark(choices=...).
  That makes mis-named or missing fields hard to diagnose. Emit a
  warning with sample_id and choices_field so the fallback is visible
  in run logs.

Signed-off-by: kanishks <kanishks@nvidia.com>
@kanishks-23
fix(byob): include data_files and field in HuggingFaceFetcher cache key
9556770 
HuggingFaceFetcher.fetch built the cache filename from (dataset_name, config, split, filters) only, so URIs that differ only in their data_files= / field= query params collided on the same cache entry. The first fetch's content was returned for every subsequent language regardless of which file was actually requested.

This silently broke per-language datasets that share a single HF repo but split content across files (e.g. IndicGenBench / FLORES_in's flores_en_<lang>_test.json, IndicGenBench / CrossSum_in's crosssum_english-<lang>_test.json). Every language's task evaluated against whichever language was fetched first.

Append `datafile-<value>` and `field-<value>` segments to the cache filename when those options are present so each (data_files, field) pair gets a distinct on-disk cache entry.

Adds a regression test that fetches three different data_files= URIs and asserts each lands in its own cache file with its own contents.

Signed-off-by: kanishks <kanishks@nvidia.com>
@kanishks-23 kanishks-23 closed this May 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@wprazuch wprazuch wprazuch approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation nemo-evaluator tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kanishks-23 @wprazuch