Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How get output from fine tuned llama3 model(trained with alpaca format dataset) in a json format ? #1744

Open
ApurvPujari opened this issue Sep 18, 2024 · 0 comments

Comments

@ApurvPujari
Copy link

ApurvPujari commented Sep 18, 2024

Is your feature request related to a problem? Please describe.
I have my own gguf file.
I was loading it like this :


from llama_cpp import Llama
llm = Llama(model_path=model_pth,
                    n_ctx=CONTEXT_SIZE,
                    chat_format="alpaca")
model_output = model(user_prompt,max_tokens=max_tokens,temperature=temperature,
      top_p=top_p,echo=False,stop=stop)
print(model_ouput)

but the "model_ouput" is of string type, and its format is not consistant. Thus I cant extract desired output from this string using a common function.

Describe the solution you'd like
I want to know a way, where I can get output in a consistant structured format.

Describe alternatives you've considered
I've tried this, but it doesnt work

llm = Llama(model_path=model_pth,
                    n_ctx=CONTEXT_SIZE,
                    chat_format="alpaca")
llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs in JSON.",
        },
        {"role": "user", "content": "Who won the world series in 2020"},
    ],
    response_format={
        "type": "json_object",
    },
    temperature=0.7,
)

Additional context
I am new to this field, any suggestion is appriciated. Thanks !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant