Skip to content

Commit 4048eb2

Browse files
authored
should fix most inference endpoints issues of version config (#226)
Updates version + allow trust remote code to launch Qwen 2 models with TGI (see slack for a bug that @meg-huggingface encountered)
1 parent 9242faf commit 4048eb2

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

src/lighteval/models/endpoint_model.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,10 +91,11 @@ def __init__(
9191
"MAX_INPUT_LENGTH": "2047",
9292
"MAX_TOTAL_TOKENS": "2048",
9393
"MODEL_ID": "/repository",
94+
"HF_MODEL_TRUST_REMOTE_CODE": "true",
9495
**config.get_dtype_args(),
9596
**config.get_custom_env_vars(),
9697
},
97-
"url": (config.image_url or "ghcr.io/huggingface/text-generation-inference:1.1.0"),
98+
"url": (config.image_url or "ghcr.io/huggingface/text-generation-inference:latest"),
9899
},
99100
)
100101
hlog("Deploying your endpoint. Please wait.")

0 commit comments

Comments
 (0)