Top_P,Top_K and Min_P arguments for ollama models

[athmanar]

Hello,
I don't understand how to set the parameters for ollama models.
This is my configuration:

config_list= [{"model": "llama3:70b-instruct", "base_url": "http://localhost:11434/v1", 'api_key': 'ollama', "price": [0, 0]}]
code_creator = autogen.AssistantAgent( name="code_creator", llm_config={ "cache_seed": 43, "config_list": config_list, "temperature": 0.9, }) #WORKS!
code_creator = autogen.AssistantAgent( name="code_creator", llm_config={ "cache_seed": 43, "config_list": config_list, "temperature": 0.9, "top_k": 40,"min_p": 0.05,"top_p": 0.95}) #DOES NOT WORK!
code_creator = autogen.AssistantAgent( name="code_creator", llm_config={ "cache_seed": 43, "config_list": config_list, "temperature": 0.9,"top_p": 0.95}) #WORKS!

It seems only top_p is supported? Am i missing something?

[ekzhu]

Please read the documentation on Ollama usage: https://microsoft.github.io/autogen/0.2/docs/topics/non-openai-models/local-ollama#api-parameters

min_p is not supported.

[athmanar]

@ekzhu Actually it is not working ..

            config_list= [{"model": "llama3:70b-instruct", "client_host": ollamakey, 'api_type': 'ollama',"temperature": 1,"top_k": 50,} ]
            autogen.AssistantAgent(name=name,llm_config={"cache_seed": cache_seed,"config_list": config_list,},system_message=system_message)

I think there is an implementation error:

    raise RuntimeError(f"Ollama exception occurred: {e}")
RuntimeError: Ollama exception occurred: Client.chat() got an unexpected keyword argument 'temperature'

Even though document mentions this is possible.

[ekzhu]

@marklysze is this something you can take a look at?

[athmanar]

@ekzhu @marklysze One fix could be adding the temperature,top_k etcg in the options dictionary.

options={'temperature':temperature,'top_k':top_k}
config_list[0]['options']=options

This does not create any issues but I am not sure if ollama is honoring these options or simple ignoring.

[marklysze]

@athmanar, would you be able to provide a simple code example with agents and messages and I'll test it

[athmanar]

@marklysze

The simple code snippet would be like this:
But please make sure ollama is running on the server at that port.

import tempfile
from autogen import ConversableAgent
from autogen.coding import LocalCommandLineCodeExecutor
temp_dir = tempfile.TemporaryDirectory()
executor = LocalCommandLineCodeExecutor( timeout=10, work_dir=temp_dir.name, )
code_executor_agent = ConversableAgent( "code_executor_agent", llm_config=False, code_execution_config={"executor": executor}, human_input_mode="ALWAYS", )
code_writer_system_message = """You are a helpful AI assistant.
Solve tasks using your coding and language skills.
In the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.
1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.
2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.
Solve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.
When using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.
If you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.
If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
When you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.
Reply 'TERMINATE' in the end when everything is done.
"""
code_writer_agent = ConversableAgent(
    "code_writer_agent",
    system_message=code_writer_system_message,
    llm_config={"config_list": [{"model": "llama3:70b-instruct","client_host": "http://localhost:11434", 'api_type': 'ollama','stream': False,'temperature':0.95} ]},
    code_execution_config=False,
)

chat_result = code_executor_agent.initiate_chat(
    code_writer_agent,
    message="Write Python code to calculate the 14th Fibonacci number.",
)

So basically this gives the error :
RuntimeError: Ollama exception occurred: Client.chat() got an unexpected keyword argument 'temperature'

[marklysze]

Thanks, let me try it out (though can't run 70b so will run a smaller model ;) )

[marklysze]

Okay, yes, Ollama has recently changed the handling of a few parameters and moved them to the options dictionary. I'll put through a PR to update this repository.

The repo for package autogen, as noted in #3936, has this update already.

[marklysze]

@athmanar, please see PR 3952, you can try replacing your ollama.py file and see if it fixes it for you. It worked for me with your code.
UPDATE: Recreating as I don't think my branch was right.

[marklysze]

Okay, so it looks I already corrected it 3 weeks ago #3593. So it's probably a version thing. So in this repo, this file is the latest one:
https://github.com/microsoft/autogen/blob/0.2/autogen/oai/ollama.py

If you're using the autogen package, which is this repo, then that is also updated.

Can you try to update your package or replace your ollama.py.

[athmanar]

@marklysze I can confirm that this now works. I replaced only the ollama file.