chatchat并发问题(均衡负载) #4424
Unanswered
blackey-endless
asked this question in
Q&A
chatchat并发问题(均衡负载)
#4424
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
请问目前chatchat是否支持模型的均衡负载，比如我在a、b两台机器上各起一个模型，然后在c机器上部署chatchat，c机器的model_config.py用online_llm_model，online_llm_model里填写a，b两台机器的服务url。然后c在接收到问答请求时，会根据a，b的资源占用情况，均衡分配请求给a还是b去推理。还是说均衡负载应该是提供模型功能的那一方去做，如果是的话有没有案例或者建议
Beta Was this translation helpful? Give feedback.
All reactions