Skip to content

Revert "[Model] Deprecate the score task (this will not affect users)." (#37537)#37726

Closed
zhewenl wants to merge 1 commit intovllm-project:mainfrom
zhewenl:auto-revert/pr-37537
Closed

Revert "[Model] Deprecate the score task (this will not affect users)." (#37537)#37726
zhewenl wants to merge 1 commit intovllm-project:mainfrom
zhewenl:auto-revert/pr-37537

Conversation

@zhewenl
Copy link
Collaborator

@zhewenl zhewenl commented Mar 21, 2026

Revert of #37537

This reverts #37537 (merge commit ed359c4).

Reason: This PR is linked to 1 new CI failure in build #57332:

  • Language Models Test (MTEB)nvidia/llama-nemotron-rerank-1b-v2 rerank MTEB score dropped marginally (diff=0.0023 vs atol=0.002), causing test_rerank_models_mteb[model_info0] to fail.

The PR changed pooler heads, activations, and scoring-related code which directly affects the reranking pipeline.


Auto-generated by CI failure analyzer.

@mergify
Copy link

mergify bot commented Mar 21, 2026

Documentation preview: https://vllm--37726.org.readthedocs.build/en/37726/

@mergify mergify bot added documentation Improvements or additions to documentation frontend v1 labels Mar 21, 2026
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request reverts the deprecation of the score task, which was causing a CI failure. The changes correctly restore the score task and its related logic across documentation, tests, and core components. The refactoring done as part of this revert, such as renaming variables for clarity in pooling heads, is a good improvement. I've found one issue where the sagemaker router is missing support for the token_embed task for scoring, which I've commented on.

(RerankRequest, (rerank, do_rerank)),
]

if "score" in supported_tasks or "embed" in supported_tasks:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The condition to enable the ScoreRequest endpoint is missing the token_embed task. The score API supports late-interaction models which use the token_embed task. This should be included to ensure full functionality on the Sagemaker endpoint, consistent with other entrypoints.

Suggested change
if "score" in supported_tasks or "embed" in supported_tasks:
if "score" in supported_tasks or "embed" in supported_tasks or "token_embed" in supported_tasks:

@noooop
Copy link
Collaborator

noooop commented Mar 21, 2026

We need this PR to unblock v2 runner. I will investigate this CI failure and fix it ASAP, please do not revert it.

@noooop
Copy link
Collaborator

noooop commented Mar 22, 2026

Close this PR as this issue should be fixed by #37775.

@noooop noooop closed this Mar 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation frontend v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants