Skip to content

Commit bb5aa7c

Browse files
raptorsunraptorsun
committed
lcore-330: complete documents about OpenAI provider
Signed-off-by: Haoyu Sun <[email protected]>
1 parent d89f7a3 commit bb5aa7c

File tree

1 file changed

+112
-56
lines changed

1 file changed

+112
-56
lines changed

‎README.md‎

Lines changed: 112 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -15,62 +15,67 @@ The service includes comprehensive user data collection capabilities for various
1515

1616
<!-- vim-markdown-toc GFM -->
1717

18-
* [Architecture](#architecture)
19-
* [Prerequisites](#prerequisites)
20-
* [Installation](#installation)
21-
* [Configuration](#configuration)
22-
* [Integration with Llama Stack](#integration-with-llama-stack)
23-
* [Llama Stack as separate server](#llama-stack-as-separate-server)
24-
* [MCP Server and Tool Configuration](#mcp-server-and-tool-configuration)
25-
* [Configuring MCP Servers](#configuring-mcp-servers)
26-
* [Configuring MCP Headers](#configuring-mcp-headers)
27-
* [Llama Stack project and configuration](#llama-stack-project-and-configuration)
28-
* [Check connection to Llama Stack](#check-connection-to-llama-stack)
29-
* [Llama Stack as client library](#llama-stack-as-client-library)
30-
* [Llama Stack version check](#llama-stack-version-check)
31-
* [User data collection](#user-data-collection)
32-
* [System prompt](#system-prompt)
33-
* [Safety Shields](#safety-shields)
34-
* [Authentication](#authentication)
35-
* [K8s based authentication](#k8s-based-authentication)
36-
* [JSON Web Keyset based authentication](#json-web-keyset-based-authentication)
37-
* [No-op authentication](#no-op-authentication)
38-
* [CORS](#cors)
39-
* [Default values](#default-values)
40-
* [Allow credentials](#allow-credentials)
41-
* [RAG Configuration](#rag-configuration)
42-
* [Example configurations for inference](#example-configurations-for-inference)
43-
* [Usage](#usage)
44-
* [Make targets](#make-targets)
45-
* [Running Linux container image](#running-linux-container-image)
46-
* [Building Container Images](#building-container-images)
47-
* [Llama-Stack as Separate Service (Server Mode)](#llama-stack-as-separate-service-server-mode)
48-
* [Llama-Stack as Library (Library Mode)](#llama-stack-as-library-library-mode)
49-
* [Verify it's running properly](#verify-its-running-properly)
50-
* [Endpoints](#endpoints)
51-
* [OpenAPI specification](#openapi-specification)
52-
* [Readiness Endpoint](#readiness-endpoint)
53-
* [Liveness Endpoint](#liveness-endpoint)
54-
* [Publish the service as Python package on PyPI](#publish-the-service-as-python-package-on-pypi)
55-
* [Generate distribution archives to be uploaded into Python registry](#generate-distribution-archives-to-be-uploaded-into-python-registry)
56-
* [Upload distribution archives into selected Python registry](#upload-distribution-archives-into-selected-python-registry)
57-
* [Packages on PyPI and Test PyPI](#packages-on-pypi-and-test-pypi)
58-
* [Contributing](#contributing)
59-
* [Testing](#testing)
60-
* [License](#license)
61-
* [Additional tools](#additional-tools)
62-
* [Utility to generate OpenAPI schema](#utility-to-generate-openapi-schema)
63-
* [Path](#path)
64-
* [Usage](#usage-1)
65-
* [Utility to generate documentation from source codes](#utility-to-generate-documentation-from-source-codes)
66-
* [Path](#path-1)
67-
* [Usage](#usage-2)
68-
* [Data Export Integration](#data-export-integration)
69-
* [Quick Integration](#quick-integration)
70-
* [Documentation](#documentation)
71-
* [Project structure](#project-structure)
72-
* [Configuration classes](#configuration-classes)
73-
* [REST API](#rest-api)
18+
- [lightspeed-stack](#lightspeed-stack)
19+
- [About The Project](#about-the-project)
20+
- [Architecture](#architecture)
21+
- [Prerequisites](#prerequisites)
22+
- [Installation](#installation)
23+
- [Run LCS locally](#run-lcs-locally)
24+
- [Configuration](#configuration)
25+
- [LLM Compatibility](#llm-compatibility)
26+
- [Set LLM provider and model](#set-llm-provider-and-model)
27+
- [Integration with Llama Stack](#integration-with-llama-stack)
28+
- [Llama Stack as separate server](#llama-stack-as-separate-server)
29+
- [MCP Server and Tool Configuration](#mcp-server-and-tool-configuration)
30+
- [Configuring MCP Servers](#configuring-mcp-servers)
31+
- [Configuring MCP Headers](#configuring-mcp-headers)
32+
- [Llama Stack project and configuration](#llama-stack-project-and-configuration)
33+
- [Check connection to Llama Stack](#check-connection-to-llama-stack)
34+
- [Llama Stack as client library](#llama-stack-as-client-library)
35+
- [Llama Stack version check](#llama-stack-version-check)
36+
- [User data collection](#user-data-collection)
37+
- [System prompt](#system-prompt)
38+
- [Safety Shields](#safety-shields)
39+
- [Authentication](#authentication)
40+
- [K8s based authentication](#k8s-based-authentication)
41+
- [JSON Web Keyset based authentication](#json-web-keyset-based-authentication)
42+
- [No-op authentication](#no-op-authentication)
43+
- [CORS](#cors)
44+
- [Default values](#default-values)
45+
- [Allow credentials](#allow-credentials)
46+
- [RAG Configuration](#rag-configuration)
47+
- [Example configurations for inference](#example-configurations-for-inference)
48+
- [Usage](#usage)
49+
- [Make targets](#make-targets)
50+
- [Running Linux container image](#running-linux-container-image)
51+
- [Building Container Images](#building-container-images)
52+
- [Llama-Stack as Separate Service (Server Mode)](#llama-stack-as-separate-service-server-mode)
53+
- [Llama-Stack as Library (Library Mode)](#llama-stack-as-library-library-mode)
54+
- [Verify it's running properly](#verify-its-running-properly)
55+
- [Endpoints](#endpoints)
56+
- [OpenAPI specification](#openapi-specification)
57+
- [Readiness Endpoint](#readiness-endpoint)
58+
- [Liveness Endpoint](#liveness-endpoint)
59+
- [Publish the service as Python package on PyPI](#publish-the-service-as-python-package-on-pypi)
60+
- [Generate distribution archives to be uploaded into Python registry](#generate-distribution-archives-to-be-uploaded-into-python-registry)
61+
- [Upload distribution archives into selected Python registry](#upload-distribution-archives-into-selected-python-registry)
62+
- [Packages on PyPI and Test PyPI](#packages-on-pypi-and-test-pypi)
63+
- [Contributing](#contributing)
64+
- [Testing](#testing)
65+
- [License](#license)
66+
- [Additional tools](#additional-tools)
67+
- [Utility to generate OpenAPI schema](#utility-to-generate-openapi-schema)
68+
- [Path](#path)
69+
- [Usage](#usage-1)
70+
- [Utility to generate documentation from source code](#utility-to-generate-documentation-from-source-code)
71+
- [Path](#path-1)
72+
- [Usage](#usage-2)
73+
- [Data Export Integration](#data-export-integration)
74+
- [Quick Integration](#quick-integration)
75+
- [Documentation](#documentation)
76+
- [Project structure](#project-structure)
77+
- [Configuration classes](#configuration-classes)
78+
- [REST API](#rest-api)
7479

7580
<!-- vim-markdown-toc -->
7681

@@ -98,9 +103,60 @@ Installation steps depends on operation system. Please look at instructions for
98103
- [Linux installation](https://lightspeed-core.github.io/lightspeed-stack/installation_linux)
99104
- [macOS installation](https://lightspeed-core.github.io/lightspeed-stack/installation_macos)
100105

106+
# Run LCS locally
107+
108+
To quickly get hands on LCS, we can run it using the default configurations provided in this repository:
109+
0. install dependencies using [uv](https://docs.astral.sh/uv/getting-started/installation/) `uv sync --group dev --group llslibdev`
110+
1. check Llama stack settings in [run.yaml](run.yaml), make sure we can access the provider and the model, the server shoud listen to port 8321.
111+
2. export the LLM token env var that Llama stack requires. for OpenAI, we set the env var by `export OPENAI_API_KEY=sk-xxxxx`
112+
3. start Llama stack server `uv run llama stack run run.yaml`
113+
4. check the LCS settings in [lightspeed-stack.yaml](lightspeed-stack.yaml). `llama_stack.url` should be `url: http://localhost:8321`
114+
5. start LCS server `make run`
115+
6. access LCS web UI at [http://localhost:8080/](http://localhost:8080/)
116+
101117

102118
# Configuration
103119

120+
## LLM Compatibility
121+
122+
Lightspeed Core Stack (LCS) supports the large language models from the providers listed below.
123+
124+
| Provider | Model | Tool Calling | provider_type | Example |
125+
| -------- | ---------------------------------------------- | ------------ | -------------- | -------------------------------------------------------------------------- |
126+
| OpenAI | gpt-5, gpt-4o, gpt4-turbo, gpt-4.1, o1, o3, o4 | Yes | remote::openai | [1](examples/openai-faiss-run.yaml) [2](examples/openai-pgvector-run.yaml) |
127+
| OpenAI | gpt-3.5-turbo, gpt-4 | No | remote::openai | |
128+
129+
The "provider_type" is used in the llama stack configuration file when refering to the provider.
130+
131+
For details of OpenAI model capabilities, please refer to https://platform.openai.com/docs/models/compare
132+
133+
134+
## Set LLM provider and model
135+
136+
The LLM provider and model are set in the configuration file for Llama Stack. This repository has a Llama stack configuration file [run.yaml](examples/run.yaml) that can serve as a good example.
137+
138+
The LLM providers are set in the section `providers.inference`. This example adds a inference provider "openai" to the llama stack. To use environment variables as configuration values, we can use the syntax `${env.ENV_VAR_NAME}`.
139+
140+
For more details, please refer to [llama stack documentation](https://llama-stack.readthedocs.io/en/latest/distributions/configuration.html#providers). Here is a list of llamastack supported providers and their configuration details: [llama stack providers](https://llama-stack.readthedocs.io/en/latest/providers/inference/index.html#providers)
141+
142+
```yaml
143+
inference:
144+
- provider_id: openai
145+
provider_type: remote::openai
146+
config:
147+
api_key: ${env.OPENAI_API_KEY}
148+
url: ${env.SERVICE_URL}
149+
```
150+
151+
The section `models` is a list of models offered by the inference provider. Attention that the field `model_id` is a user chosen name for referring to the model locally, the field `provider_model_id` refers to the model name on the provider side. The field `provider_id` must refer to one of the inference providers we defined in the provider list above.
152+
153+
```yaml
154+
models:
155+
- model_id: gpt-4-turbo
156+
provider_id: openai
157+
model_type: llm
158+
provider_model_id: gpt-4-turbo
159+
```
104160

105161

106162
## Integration with Llama Stack

0 commit comments

Comments
 (0)