Support custom llm

[Sardhendu]

Custom Model and Global Dict ModelRegistry support

• ModelRegistry: A Dictionary object with helper method and model info

      from autolabel.models import MODEL_REGISTRY
      print(MODEL_REGISTRY)  # Lists all the provider and models in a table format for better viewability
  

• Langchain imports done under constructor
• Methods implemented to register all predefined LLM class.
• Registering a custom Model.

  class MyCustommodel(BaseModel):
      def __init__(self, config: AutolabelConfig, cache: BaseCache = None) -> None:
          super().__init__(config, cache)
  
      def _label(self,):
          pass
  
      def get_cost(self,):
          pass
  
      def returns_token_probs(self,):
          pass
  
  
  MODEL_REGISTRY.register(model_name="my_custom_llm", model=MyCustommodel)
  

[Sardhendu]

@nihit . instantiating ModelRegistry output a global dictionary MODEL_REGISTRY. This class is just.a helper to view the list of providers and models. Let me know your thoughts about it. Rest of the code should be self explanatory.

NOTE: [Its still pending]

Please skim through it and suggest changes, if needed.

[rajasbansal]

did we move this in order to prevent import errors? Should we catch import errors in that case and return the command to install the extras related to this model?

[Sardhendu]

This was done to mimic the imports as part of previously deployed code. If langchain is a base dependency it makes more sense to have top level imports.

Agreed that we should be catching import errors during test cases.

[rajasbansal]

we can keep this out? langchain is a dependency of the library

[rajasbansal]

can keep this out?

[Sardhendu]

@rajasbansal With the updated code below is how we register the model.

register_model(name="custom", model_cls=MyCustommodel)

Note: The name has to be custom

[rajasbansal]

No need to do this in functions, we can add to the dictionary directly,

MODEL_REGISTRY = {
ModelProvider.HUGGINGFACE_PIPELINE: HFPipelineLLM,
ModelProvider.GOOGLE: PaLMLM
...
}

[rajasbansal]

lets move this to inside the model so that we don't fail if openai is not installed

[rajasbansal]

lets move this

[rajasbansal]

same: move

[rajasbansal]

lgtm. Thanks for the changes!