edit_predictions: OpenAI-compatible endpoint - #49388
Conversation
This adds a new edit prediction provider that uses the standard OpenAI
chat completions API (`/v1/chat/completions`), enabling local inference
runtimes like vLLM, TGI, llama.cpp, and LMStudio to serve edit
predictions.
The provider reuses the same Zeta and FIM prompt logic as the Ollama
provider (shared helpers are now `pub(crate)`), but sends requests as
`open_ai::Request` and parses `open_ai::Response`. Auth is optional,
supporting both a settings value and the
`OPENAI_COMPATIBLE_EDIT_PREDICTION_API_KEY` environment variable.
Configurable via settings:
{
"edit_predictions": {
"provider": "openai_compatible",
"openai_compatible": {
"api_url": "http://localhost:8000/v1",
"model": "zeta"
}
}
}
cla-bot
Bot
added
the
cla-signed
SomeoneToIgnore
added
the
area:ai
- Rename edit prediction provider module file to `open_ai_compatible.rs` and update references - Rename settings UI renderer helper to `render_open_ai_compatible_provider` - Avoid potential panic in Ollama FIM prompt construction by using `split_at_checked` - Minor log tag consistency for OpenAI-compatible provider
|
So far I'm struggling to test this with zeta on vllm. No requests seem to reach vllm for some reason, though curls to it do get a response back. I'm on a Macbook Air M2 24G, and first time using vllm on it. I got vllm to run with a limited On Zed's settings I'm also limiting output tokens: The implementation is still working with Ollama, so I'm assuming this is something to do with my vllm setup and my local env limitations. |
|
Unfortunately I've decided to close this. Since I posted the comment saying we'd be open to contributions here, we received a large number of upvotes on #49346. Given that context we've decided that the right thing to do here is turn the OpenAI compatible support into a fully-fledged feature and do some internal simplifications while we're at it to handle zeta and other fim models cleanly between the zed hosted offering, ollama, and OpenAI compatible endpoints. Given that this PR is going in a different direction, we're going to build it ourselves rather than branching off from your work. I appreciate the contribution, sorry for the change in direction. We should have a PR up to solve this in the next day or two |
|
No worries, makes sense @probably-neb. Always a learning experience, and I'm curious to see what this new implementation will look like |
|
Thanks for understanding! The updated PR is up at #49554 |
3 participants
No issue available - this takes up the second suggestion made at #24859 (comment) - cc @probably-neb 👋
Still testing and refining, but could use feedback. So far I can see predictions when using qwen2.5-coder:3b-base on Ollama
Release Notes: