Adding voyage-4-large, voyage-4 and voyage-4-lite

[fzoll]

The new models will be released 2026-01-15

related to: embeddings-benchmark/results#391

[Samoed]

Should we wait until public release?

[fzoll]

@Samoed @KennethEnevoldsen Would it be possible to wait with the code change but merge the results before the public release?

[bflhc]

Thanks for adding the voyage-4-large and voyage-4-lite models.

I’d like to raise a couple of points for clarity and fairness regarding private evaluations on RTEB:

1. Fair access to private evaluation:
   RTEB’s hybrid approach of using both open and private datasets is intended to provide an impartial measure of generalization to unseen data.
   It would be good to clarify how private eval results are being obtained for voyage models before their public release, given that model URLs are not yet accessible. Releasing results before models are publicly available may create an uneven situation — community members with publicly available models (like Octen-Embedding-8B) are still waiting for private evaluation.

2. Unpublished models and result merging:
   The voyage PR indicates the models will be released on 2026-01-15, and the proposer asks whether results can be merged before that release. Since the model artifacts aren’t currently accessible, it seems premature to merge results — this could conflict with the usual expectation that results correspond to reproducible models.

3. Request for Octen-Embedding-8B private evaluation:
   In Issue #3881 we’ve requested private evaluation for the new Octen-Embedding-8B version (with strong public performance and expected private gains). That request hasn’t been addressed yet.

Could the maintainers please provide an update on the feasibility or schedule for running that private evaluation? Understanding this will help maintain transparency and fairness for all contributors.

[KennethEnevoldsen]

@bflhc thanks for posting this - I have moved it to a seperate issue so that we can continue discussion there. Though I realize that these are related

[KennethEnevoldsen]

@fzoll I seem to get an error when I run the following code:

import mteb

model = mteb.get_model("voyageai/voyage-4")
task = mteb.get_task("LccSentimentClassification")
mteb.evaluate(model, task)

It seems like the tokenizer might not be public:

Traceback (most recent call last):
  File "/Users/au561649/Github/mteb/.venv/lib/python3.13/site-packages/huggingface_hub/utils/_http.py", line 402, in hf_raise_for_status
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/au561649/Github/mteb/.venv/lib/python3.13/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/voyageai/voyage-4/resolve/main/tokenizer.json

[KennethEnevoldsen]

I couldn't run the implementation (see above), I suspect you just need to make the HF repo public. Otherwise the metadata looks fine.

[fzoll]

@KennethEnevoldsen The tokenizer for voyage-4-large and voyage-4-lite are publicly available now, can you please check? (have to wait for the voyage-4, so i reverted that change)

[KennethEnevoldsen]

Updated code snippet to:

import mteb

model_names = ["voyageai/voyage-4-lite"]
for name in model_names:
    model = mteb.get_model(name)
    task = mteb.get_task("AILAStatutes")
    res = mteb.evaluate(model, task)
    score = res.task_results[0].get_score()
    print(f"Model: {name}, AILAStatutes Score: {score}")

which results in:

Model: voyageai/voyage-4-lite, AILAStatutes Score: 0.48477

Which matches the reported: .4849 within our error bound.

I also tested that both models ran without issue on the above task LccSentimentClassification.

[KennethEnevoldsen]

@Samoed asking for an extra review here due to the concern. The model is accessed through the public API.

[Samoed]

You can add to older models superseded_by, but otherwise looks good

[fzoll]

@KennethEnevoldsen @Samoed Is this PR good to be merged? Anything else missing?

[KennethEnevoldsen]

@fzoll added superseeded by. Assuming it is correct.