Refinement: setting batch_size for different models #212
Conversation
riboyuan99
requested review from
CambioML,
goldmermaid,
SayaZhang,
CluckRookie and
SeisSerenata
as code owners
|
Will format the code shortly |
|
Please reformat your code to resolve build failure https://github.com/CambioML/uniflow-llm-based-pdf-extraction-text-cleaning-data-clustering/actions/runs/8164445774/job/22319716558?pr=212 |
|
Could you please also check all notebook regarding whether you need to make corresponding changes as well as cookbook repo? Get @jojortz to review the cookbook repo. |
Noted |
| if not batch_size: | ||
| batch_size = self._config.model_config.get( | ||
| "num_thread", 1 | ||
| ) # pylint: disable=no-member | ||
|
|
There was a problem hiding this comment.
please add comment regarding why we are hacking this way, especially the logic we were talking about regarding local hosted model and proprietary models.
uniflow/op/model/model_config.py
Outdated
| @@ -26,7 +26,7 @@ class GoogleModelConfig(ModelConfig): | |||
| candidate_count: int = 1 | |||
| num_thread: int = 1 | |||
| # this is not real batch inference, but size to group for thread pool executor. | |||
| batch_size: int = 1 | |||
| # batch_size: int = 1 | |||
There was a problem hiding this comment.
you should just remove them and the same for all below. Then, check again to make sure all batch_size is removed for local hosted model.
…y 2. If batch_size is not found, use num_thread instead
Since Google's model, OpenAI's model, and Azure OpenAI's model don't support batching locally, we will instead use
num_threadto "fake batch" the data.