Skip to content

Docs update #509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 9, 2025
Merged

Docs update #509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c22d57
use npm as deps manager and vite as bundler
ngxson Nov 30, 2024
a1ff316
update XTC docs
saood06 Jun 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions examples/main/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,13 +241,11 @@ Example usage: `--mirostat 2 --mirostat-lr 0.05 --mirostat-ent 3.0`

### XTC Sampling (Exclude Top Choices)

The function of this sampler is conrolled by `--xtc-probability` and `--xtc-threshold`. `--xtc-probability` takes values between
0 and 1 (<=0 turns this sampler off) and defines the probability for randomly invoking the sampler. `--xtc-threshold`
defines the token probability threshold. Tokens with probability greater than this threshold will be excluded from the sampling.
The sampler is turned off for `threshold > 0.5`.
- --xtc-probability p: xtc probability (default: 0.0, maximum: 1.0) => (p*100)% chance to invoke the sampler.
- --xtc-threshold t : xtc threshold (default: 1.0 minimum : 0.0) => (t*100)% is the lower-bound for what probability is needed for a token to be considered a "Top choice", and only the lowest probability top choice is kept, all others are removed.

The sampler is turned off for `threshold > 0.5`, and `--xtc-probability <=0`.

- --xtc-probability p: xtc probability (default: 0.0 => disabled)
- --xtc-threshold t : xtc threshold (default: 1.0 => disabled)

### Top-n-sigma Sampling

Expand Down
31 changes: 31 additions & 0 deletions examples/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,37 @@ logging:
cmake --build build --config Release -t llama-server
```

## Web UI

The project includes a web-based user interface that enables interaction with the model through the `/chat/completions` endpoint.

The web UI is developed using:
- `vue` framework for frontend development
- `tailwindcss` and `daisyui` for styling
- `vite` for build tooling

A pre-built version is available as a single HTML file under `/public` directory.

To build or to run the dev server (with hot reload):

```sh
# make sure you have nodejs installed
cd examples/server/webui
npm i

# to run the dev server
npm run dev

# to build the public/index.html
npm run build
```

NOTE: if you are using the vite dev server, you can change the API base URL to llama.cpp. To do that, run this code snippet in browser's console:

```js
localStorage.setItem('base', 'http://localhost:8080')
```

## Quick Start

To get started right away, run the following command, making sure to use the correct path for the model you have:
Expand Down