thread 'tokio-runtime-worker' panicked at core/src/queue.rs:72:14

[michaelfeil]

System Info

docker run -it --rm -p 7997:7997 --gpus all ghcr.io/huggingface/text-embeddings-inference:hopper-1.6
--model-id BAAI/bge-large-en-v1.5 --port 7997 --max-client-batch-size 8000 --max-batch-size 64 --max-concurrent-requests 512

revision: None, tokenization_workers: Some(8), dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: Some(32), max_client_batch_size: 8000, auto_truncate: true, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "xxx", port: 7997, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }

2025-01-13T22:43:09.523081Z  INFO openai_embed{total_time="3.53211965s" tokenization_time="1.65615ms" queue_time="504.499706ms" inference_time="33.584808ms"}: text_embeddings_router::http::server: router/src/http/server.rs:1260: Success
2025-01-13T22:43:09.792384Z  INFO openai_embed{total_time="3.801234069s" tokenization_time="1.672146ms" queue_time="503.120879ms" inference_time="33.486594ms"}: text_embeddings_router::http::server: router/src/http/server.rs:1260: Success
2025-01-13T22:43:10.033664Z  INFO openai_embed{total_time="4.041983871s" tokenization_time="1.643926ms" queue_time="504.332069ms" inference_time="33.626412ms"}: text_embeddings_router::http::server: router/src/http/server.rs:1260: Success
2025-01-13T22:43:20.890172Z  INFO embed{total_time="128.536668ms" tokenization_time="2.85406ms" queue_time="58.577982ms" inference_time="9.716932ms"}: text_embeddings_router::http::server: router/src/http/server.rs:714: Success
thread 'tokio-runtime-worker' panicked at core/src/queue.rs:72:14:
Queue background task dropped the receiver or the receiver is too behind. This is a bug.: "Full(..)"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Workload:

• switching between 1min break, 1 validation request (using /embed) 2 min throughput testing benchmark (/openai_embed`)

Result:

• queue.Full error raised here:

    pub fn append(&self, entry: Entry) {
        // Send append command to the background task managing the state
        // Unwrap is safe here
        self.queue_sender
            .try_send(QueueCommand::Append(Box::new(entry), Span::current()))
            .expect("Queue background task dropped the receiver or the receiver is too behind. This is a bug.");
    }

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Expected behavior

[eli-persona]

Also seeing this concurrency control issue. We're writing up some findings now, and will create another issue with them (linking here)

[cdpierse]

Hi there, we've been running into what looks to be a similar issue for the image ghcr.io/huggingface/text-embeddings-inference:89-1.7.2 with the below config. Similar to @michaelfeil we are hitting /openai_embed for all our requests.

I was wondering if there has been any progress on this issue since last raised ?
Config:

AUTO_TRUNCATE:            true                                                                                                                                                                              
DTYPE:                    float16                                                                                                                                                                            
MAX_BATCH_REQUESTS:       32                                                                                                                                                                                
MAX_BATCH_TOKENS:         102400                                                                                                                                                                             
MAX_CLIENT_BATCH_SIZE:    200                                                                                                                                                                                
MAX_CONCURRENT_REQUESTS:  64                                                                                                                                                                                 
MODEL_ID:                 Snowflake/snowflake-arctic-embed-m-v1.5                                                                                                                                            
PAYLOAD_LIMIT:            10000000                                                                                                                                                                           

Stack trace:

 {"timestamp":"2025-06-17T16:25:51.544416Z","level":"INFO","message":"Success","target":"text_embeddings_router::http::server","filename":"router/src/http/server.rs","line_number":1285,"span":{"inference_time":"1.9134ms","queue_time":"3.012713ms","tokenization_time":"270.806µs","total_time":"5.235928ms","name":"openai_embed"},"spans":[{"inference_time":"1.9134ms","queue_time":"3.012713ms","tokenization_time":"270.806µs","total_time":"5.235928ms","name":"openai_embed"}]}
 {"timestamp":"2025-06-17T16:25:51.544439Z","level":"INFO","message":"Success","target":"text_embeddings_router::http::server","filename":"router/src/http/server.rs","line_number":1285,"span":{"inference_time":"4.410987ms","queue_time":"1.321567ms","tokenization_time":"842.918µs","total_time":"9.701003ms","name":"openai_embed"},"spans":[{"inference_time":"4.410987ms","queue_time":"1.321567ms","tokenization_time":"842.918µs","total_time":"9.701003ms","name":"openai_embed"}]}
 {"timestamp":"2025-06-17T16:25:51.544436Z","level":"INFO","message":"Success","target":"text_embeddings_router::http::server","filename":"router/src/http/server.rs","line_number":1285,"span":{"inference_time":"1.9134ms","queue_time":"1.683494ms","tokenization_time":"281.216µs","total_time":"3.938022ms","name":"openai_embed"},"spans":[{"inference_time":"1.9134ms","queue_time":"1.683494ms","tokenization_time":"281.216µs","total_time":"3.938022ms","name":"openai_embed"}]}
 
 thread 'tokio-runtime-worker' panicked at core/src/queue.rs:72:14:
 Queue background task dropped the receiver or the receiver is too behind. This is a bug.: "Full(..)"
 stack backtrace:
 {"timestamp":"2025-06-17T16:25:51.557093Z","level":"INFO","message":"Success","target":"text_embeddings_router::http::server","filename":"router/src/http/server.rs","line_number":1285,"span":{"inference_time":"1.603446ms","queue_time":"1.189409ms","tokenization_time":"342.067µs","total_time":"8.056518ms","name":"openai_embed"},"spans":[{"inference_time":"1.603446ms","queue_time":"1.189409ms","tokenization_time":"342.067µs","total_time":"8.056518ms","name":"openai_embed"}]}
    0:     0x56450e03d44f - <unknown>
    1:     0x56450dc29983 - <unknown>
    2:     0x56450e03cba2 - <unknown>
    3:     0x56450e03d2b3 - <unknown>
    4:     0x56450e03c977 - <unknown>
    5:     0x56450e079468 - <unknown>
    6:     0x56450e0793c9 - <unknown>
    7:     0x56450e07986c - <unknown>
    8:     0x56450d48306f - <unknown>
    9:     0x56450d483415 - <unknown>
   10:     0x56450e1ddcc7 - <unknown>
   11:     0x56450d8671b9 - <unknown>
   12:     0x56450d86309a - <unknown>
   13:     0x56450d862cfa - <unknown>
   14:     0x56450d892a75 - <unknown>
   15:     0x56450d891d59 - <unknown>
   16:     0x56450d9b22b3 - <unknown>
   17:     0x56450d817c00 - <unknown>
   18:     0x56450d9c7166 - <unknown>
   19:     0x56450d8185a2 - <unknown>
   20:     0x56450d9c7166 - <unknown>
   21:     0x56450d8185a2 - <unknown>
   22:     0x56450d9c7166 - <unknown>
   23:     0x56450d8185a2 - <unknown>
   24:     0x56450d9c7166 - <unknown>
   25:     0x56450d817e4d - <unknown>
   26:     0x56450d9c7166 - <unknown>
   27:     0x56450d8183ca - <unknown>
   28:     0x56450d9005a8 - <unknown>
   29:     0x56450d85d3c2 - <unknown>
   30:     0x56450d81951b - <unknown>
   31:     0x56450d9c7166 - <unknown>
   32:     0x56450d8185a2 - <unknown>
   33:     0x56450d9c7166 - <unknown>
   34:     0x56450d81880e - <unknown>
   35:     0x56450d9c7166 - <unknown>
   36:     0x56450d8469ec - <unknown>
   37:     0x56450d93e722 - <unknown>
   38:     0x56450d820241 - <unknown>
   39:     0x56450d95857f - <unknown>
   40:     0x56450e2bc310 - <unknown>
   41:     0x56450e2bf510 - <unknown>
   42:     0x56450e2afed4 - <unknown>
   43:     0x56450e2b3deb - <unknown>
   44:     0x56450e07a5fb - <unknown>
   45:     0x7feaed472ac3 - <unknown>
   46:     0x7feaed504a40 - <unknown>
   47:                0x0 - <unknown>

[michaelfeil]

Yeah.

1. add n+1 permits for n allowed requests
2. more likley with higher number of permits

Permits are controlled with --max-client-request-number-setting

[cdpierse]

Thanks @michaelfeil so are you saying setting MAX_CLIENT_BATCH_SIZE to say 201 while the API controls the true batch size to a limit of 200 has helped mitigate this for you ?

[cdpierse]

We were able to replicate this bug in a test env by setting max_concurrent_requests to a very low number and then saturating the server with many small requests.

While the expected behaviour would be for the server to respond to the client with a model is overloaded error we found that saturating the server many requests causes a panic, resulting in this exception to be raised:

text-embeddings-inference/core/src/queue.rs

Line 71 in d1d65cf

.try_send(QueueCommand::Append(Box::new(entry), Span::current()))

Thanks again @michaelfeil for pointing this out to us.