Qwen3 4B model embs only 1024 dims (not 2560)?

[deklanw]

System Info

HF inference endpoint with custom container ghcr.io/huggingface/text-embeddings-inference:1.7.2 and deployed Qwen/Qwen3-Embedding-4B with task "sentence-embeddings" on a single L4 machine from AWS.

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Just deploy a HF endpoint with Qwen/Qwen3-Embedding-4B and using ghcr.io/huggingface/text-embeddings-inference:1.7.2 image and make a test embedding.

Expected behavior

The embeddings I'm getting back are dim 1024 but this model should support up to 2560?

[pocman]

Overall, the readme should probably have a section regarding embedding size and the fact that Matryoshka Representation Learning is not supported.

[pocman]

I deployed Qwen3 4B using code from main branch and I do have 2560 dims.
2adbd12

[MohannadEhabBarakat]

I have the same issue with latest release

[MohannadEhabBarakat]

Sorry for the confusion it was my mistake

[alvarobartt]

Hey @deklanw could you try with 1.7.3 instead? That should resolve some of the issues around Qwen3, see https://github.com/huggingface/text-embeddings-inference/releases/tag/v1.7.3.

Also note that as per another recent report (see #644), we'll be doing another small patch release with a potential fix for it on Monday i.e., tomorrow, so stay tuned for 1.7.4 with all the latest changes.

[deklanw]

Same issue with 1.7.4. @alvarobartt

However, I notice that the default custom container for this model is now mfuntowicz/endpoints-embeddings-qwen3:v1.0.1. With this container the dimensionality is correct. No idea what this container is how or it relates to TEI?

[alvarobartt]

Oh fair, I think that's because Qwen3 is still lacking Matryoska Representation Learning (MRL) which lets you define the dimensions for the embeddings, see more about it #676. In any case I'll investigate the issue and compare TEI implementation against Sentence Transformers!

Alternatively, as per mfuntowicz/endpoints-embeddings-qwen3:v1.0.1 that's a container we're using for a new experiment around Inference Endpoints, more at https://github.com/huggingface/hfendpoints, but still experimental, Qwen3 models are release with it leveraging the Sentence Transformers backend, so that might explain the differences.

[alvarobartt]

Hey again @deklanw I just tried with main and it works just fine and it returns embeddings of length 2560 for Qwen/Qwen3-Embedding-4B, could you please let me know how you are running it? Could you make sure to reproduce with the latest version?

e.g. for reproducing I ran cargo run --release --features candle-cuda,dynamic-linking,http --no-default-features -- --model-id Qwen/Qwen3-Embedding-4B --dtype float16 --max-batch-tokens 512 on an instance with an NVIDIA L40S GPU, and it worked just fine! Also same for --model-id Qwen/Qwen3-Embedding-0.6B which produced embeddings with dimension 1024, and for --model-id Qwen/Qwen3-Embedding-8B which produced embeddings with dimension 4096, by default, as per https://huggingface.co/Qwen/Qwen3-Embedding-0.6B#qwen3-embedding-series-model-list.

[deklanw]

I simply create an endpoint with HF endpoint. Custom container: ghcr.io/huggingface/text-embeddings-inference:1.7.4 deployed on an AWS L4.

Looking at the logs just now, though, I see something suspicious:

Args { model_id: "BAA*/***-*****-**-v1.5", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hf_token: None, hostname: "r-borgcollectivegmbh-qwen3-embedding-4b-xze-l4hycd31-2a1c75h72j", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/repository/cache"), payload_limit: 2000000, api_key: None, json_output: true, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }

That model ID is incorrect? Seems to be BAAI/bge-large-en-v1.5 (which does have 1024 dims!), but I definitely deployed Qwen 3 4B.

[alvarobartt]

Hmm that's odd indeed, let me ping @ErikKaum and investigate further, thanks a lot for the report!

It's because the default --model-id on Text Embeddings Inference (TEI) comes with a default value being BAAI/bge-large-en-v1.5 (something we discussed with @Narsil that may make sense to remove in the next minor release), and apparently on Inference Endpoints when running a custom container the MODEL_ID environment variable is not set but instead HF_MODEL_ID meaning that the --model-id won't have a value, and the default will be used instead.

So on, setting the environment variable MODEL_ID to Qwen/Qwen3-Embedding-4B solves your issue, apologies in advance 🤗

[deklanw]

Oh. Thanks. Since Inference Endpoints knows what model I'm deploying why is that not set for me?

[alvarobartt]

@deklanw it makes sense, and apologies for that! It seems that for custom containers i.e., when you select "Custom" the environment variables are indeed set in advance, but those are identified as HF_MODEL_ID, HF_REPOSITORY, etc. and only handled "properly" when you specifically select the "Text Embeddings Inference" option (not available for this model but we'll make sure to enable). In any case, this has already been reported internally, and we'll discuss with the team.

Is not that the container doesn't know which model, is just that the environment variable naming we have in there for custom containers comes with the HF_ prefix, which is preventing it to be captured by either Text Generation Inference or Text Embeddings Inference, due to those values expecting e.g. MODEL_ID instead of HF_MODEL_ID.

Hope this can help clarify a bit 🤗