Support using ollama as an inline_completion_provider

[aitorpazos]

Check for existing issues

• Completed

Describe the feature

I am successfully using my local ollama models using assistant panel.

I would love to be able to use them as well as an inline_completion_provider.

Currently, only none, copilot or supermaven values are supported.

If applicable, add mockups / screenshots to help present your vision of the feature

No response

[MatejLach]

Some considerations that come to mind regarding this, keeping in mind the current support for Ollama:

In contrast to something like GitHub Copilot, whose entire purpose is to provide inline completions, some of the most popular models used with Ollama, such as llama3.1 DO NOT support inline completion.
For those that do, such as deepseek-coder-v2, you can make a POST request to /api/generate like:

{
  "model": "deepseek-coder-v2:latest",
  "prompt": "time := time.",   
  "suffix": "    return time;",
  "options": {
    "temperature": 0
  },
  "keep_alive": -1,
  "stream": false
}

And get back a response such as:

{
  "model": "deepseek-coder-v2:latest",
  "created_at": "2024-09-07T14:28:17.013718016Z",
  "response": "Now().Unix()\n",   // our inline completion, inserted between prompt and suffix
  "done": true,
  "done_reason": "stop",
  // metadata fields omitted 
}

For models that do not support inline completions the above request results in i.e.:

{
  "error": "llama3.1:latest does not support insert"
}

However it would be desirable to use say llama3.1 for chat while using deepseek-coder-v2 for inline completion at the same time, therefore the list of Olllama inline completion models should be separate from chat models.

Another consideration is how much sense it would make to support remote Ollama instances for inline completions. , I've got Ollama running both locally on my laptop and on a server I've got on my LAN to get access to more powerful models.
Anecdotally, the difference between local response times (~100ms, vs the server ~300ms), mean that using the LAN server for inline completions is probably impractical vs using it for chat is just fine.
It also means that it would be nice if the chat Ollama provider vs the inline completion Ollama provider did not share the same base URL.

[skewty]

This repo was mentioned in #14134.
I am including it here so it doesn't get forgotten and for easier reference.

---

Proxy that allows you to use ollama as a copilot like Github copilot

https://github.com/bernardo-bruning/ollama-copilot

Written in Go

[xmaayy]

@skewty Seems like that wont work?
#6701

[tlvenn]

As mentioned in #16030 to address your concern @MatejLach, we should be able to configure an autocomplete model along side a chat model.

Copy pasting from the issue:

Heres an example of the config.json in Continue.dev to change autocomplete model:

  "tabAutocompleteModel": {
    "title": "Tab Autocomplete Model",
    "provider": "ollama",
    "apiBase": "http://localhost:11434",
    "model": "deepseek-coder:6.7b",
    "contextLength": 8192
  },
  "tabAutocompleteOptions": {
    "prefixPercentage": 0.5,
    "maxPromptTokens": 4096
  },

[navidRashik]

@tlvenn this config is for continue.dev and this extention is not present in zed editor now that how would that work in zed?

[F1shez]

I just started testing zed and I think it's a really necessary and important task for to transition from vscode with continue.dev

[jorge-menjivar]

This feature is a must if I am going to move from Cursor. Github copilot is not good enough so I at least want to have more options.

[mbitsnbites]

Another consideration is how much sense it would make to support remote Ollama instances for inline completions

Not sure if there are any specific concerns with ollama, but generally speaking I think it makes perfect sense to support remote instances. For instance I sometimes use an old quad-core Celeron laptop (with 4 GB RAM and no GPU) for coding. It's completely useless for LLM inference, but I have a 12-core Linux server with 32GB RAM and a GTX 1080 GPU in my LAN and it serves up 7B models in llama.cpp without breaking a sweat. I'm also quite confident that my LAN does not add any noticeable lag. I for one would prefer running models on a locally hosted remote server.

[mcmacker4]

Another consideration is how much sense it would make to support remote Ollama instances for inline completions

Not sure if there are any specific concerns with ollama, but generally speaking I think it makes perfect sense to support remote instances.

I think, given that ollama is an HTTP API, the specifics of where you host your ollama instance is not really of Zed's concern, right?
Wether you pont Zed to a localhost URL or an IP three countries over, Zed's only ask is that the API is accessible.