object has no attribute 'set_run_config'

[nprime496]

Hi, I'm encountering an issue with my code while using a chat model, and I'm getting the following error:

self.llm.set_run_config(run_config)
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LLMRAGEvaluator' object has no attribute 'set_run_config'

LLMRAGEvaluator is a class I created.

Here's the code snippet I'm working with:

from deepeval import evaluate
from deepeval.metrics.ragas import RagasMetric
from deepeval.test_case import LLMTestCase
from langchain_ollama import ChatOllama

from typing import List
from deepeval.models.base_model import DeepEvalBaseLLM

class LLMRAGEvaluator(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def generate(self, prompt: str) -> str:
        return self.model.invoke(prompt).content
        
    async def a_generate(self, prompt: str) -> str:
        response = await self.model.ainvoke(prompt)
        return response.content
  
    def get_model_name(self):
        return "ollama"

ollama_evaluator = LLMRAGEvaluator(model=ChatOllama(model="llama3.1:8b"))

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = RagasMetric(threshold=0.5, model=ollama_evaluator)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score)

# or evaluate test cases in bulk
evaluate([test_case], [metric])

Can you suggest any modifications or solutions to make the evaluator work without this error?

[penguine-ip]

Yes @nprime496 the ragas metric supports langchain chatmodels only, not the usual DeepEvalBaseLLM for native metrics on deepeval: https://docs.confident-ai.com/docs/metrics-ragas#required-arguments

also support for ragas v0.2 is coming out in a few hours, currently you'll have to stick to <0.2