Code

tokenizer : special token handling#3538

Code

Merged

ggerganov merged 11 commits intoggml-org:masterggml-org/llama.cpp:masterfrom

staviq:specialtokensstaviq/llama.cpp:specialtokensCopy head branch name to clipboard

Oct 17, 2023

Commits

Commits on Oct 8, 2023

• Rewrite special token handling from #1931

  staviq

  committed
  b592c70

  Copy full SHA for b592c70

Commits on Oct 10, 2023

• shorten param name, add st verification by type

  staviq

  committed
  fc634d8

  Copy full SHA for fc634d8

Commits on Oct 11, 2023

• use offsets instead of copy by substr

  staviq

  committed
  29e6b46

  Copy full SHA for 29e6b46

• Merge branch 'master' into specialtokens

  staviq

  authored
  eac5f54

  Copy full SHA for eac5f54

• formatting, remove copying iterator on delete

  staviq

  committed
  f7b1205

  Copy full SHA for f7b1205

Commits on Oct 12, 2023

• Merge branch 'master' into HEAD

  ggerganov

  committed
  04ac055

  Copy full SHA for 04ac055

• llama : normalize code-style

  ggerganov

  committed
  5c6b2be

  Copy full SHA for 5c6b2be

• swift fix

  staviq

  committed
  0f1c569

  Copy full SHA for 0f1c569

• print pfx/sfx if verb, main: split pfx input sfx

  staviq

  committed
  5974d61

  Copy full SHA for 5974d61

• dont add space when using special tokens

  staviq

  committed
  1c28116

  Copy full SHA for 1c28116

Commits on Oct 17, 2023

• minor : comment + spacing

  ggerganov

  authored
  fc82541

  Copy full SHA for fc82541