Skip to content

refactor: kv cache manager repo #570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vMaroon merged 27 commits into from
Jan 21, 2026
Merged

refactor: kv cache manager repo #570

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1c15fd2
refactor: kv cache manager repo name
sagearc Jan 19, 2026
2683a46
Merge branch 'main' into rename-llmd-kv-cache
sagearc Jan 19, 2026
2facce6
go mod tidy
sagearc Jan 19, 2026
5755ebd
fetch kv cache upstream instead of my fork
sagearc Jan 19, 2026
824c89b
revert dockerfile to fetch kv cache manager from upstream instead of …
sagearc Jan 19, 2026
58f6a4b
update chat preprocessing structs
sagearc Jan 19, 2026
b89abce
update kv cache manager version
sagearc Jan 20, 2026
a26d4b1
refactor kvblock.Key to kvblock.BlockHash
sagearc Jan 20, 2026
2e43da1
add context
sagearc Jan 20, 2026
4e44531
add parent block key
sagearc Jan 20, 2026
3a9a6d9
refactor encode
sagearc Jan 20, 2026
9c820cf
validate model name
sagearc Jan 20, 2026
4edcd32
run setup.sh
hyeongyun0916 Jan 20, 2026
2a5a77e
clone vllm into build
sagearc Jan 20, 2026
b63ef84
edit
hyeongyun0916 Jan 20, 2026
31eefb9
edit lint
hyeongyun0916 Jan 20, 2026
9c943e7
Merge commit 'f272b8549a9d870357aa9900ed972836136bc019' into rename-l…
hyeongyun0916 Jan 20, 2026
a50eb9d
delete fetch-python-wrapper.sh
hyeongyun0916 Jan 20, 2026
b9fd43d
edit git workflow
hyeongyun0916 Jan 20, 2026
d7653bf
edit
hyeongyun0916 Jan 21, 2026
05ba63c
Merge branch 'main' into rename-llmd-kv-cache
hyeongyun0916 Jan 21, 2026
2e59a41
refactor TokenProcessorConfig in config
sagearc Jan 21, 2026
cd74019
fix kv cache repo name in docker file
sagearc Jan 21, 2026
7861c04
fix e2e tests
vMaroon Jan 21, 2026
c5f12b0
Merge commit '4884fc207f3bcafc8adb8fe705bbe114c6f9e9d4' into rename-l…
hyeongyun0916 Jan 21, 2026
886cb07
add ignore
hyeongyun0916 Jan 21, 2026
1848a6a
update architecture docs
sagearc Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions Dockerfile.epp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,20 @@ COPY pkg/ pkg/

RUN go mod download

# Copy Python wrapper and requirements from llm-d-kv-cache-manager dependency
# Copy Python wrapper and requirements from llm-d-kv-cache dependency
# Extract version dynamically and copy to a known location
# We need to keep llm-d-kv-cache-manager as go module path is kept the old name
RUN KVCACHE_MANAGER_VERSION=$(go list -m -f '{{.Version}}' github.com/llm-d/llm-d-kv-cache-manager) && \
# We need to use the replacement module path in the cache
RUN KVCACHE_MANAGER_VERSION=$(go list -m -f '{{.Replace.Version}}' github.com/llm-d/llm-d-kv-cache) && \
KVCACHE_MANAGER_PATH=$(go list -m -f '{{.Replace.Path}}' github.com/llm-d/llm-d-kv-cache) && \
mkdir -p /workspace/kv-cache && \
cp /go/pkg/mod/github.com/llm-d/llm-d-kv-cache-manager@${KVCACHE_MANAGER_VERSION}/pkg/preprocessing/chat_completions/render_jinja_template_wrapper.py \
cp /go/pkg/mod/${KVCACHE_MANAGER_PATH}@${KVCACHE_MANAGER_VERSION}/pkg/preprocessing/chat_completions/render_jinja_template_wrapper.py \
/workspace/kv-cache/render_jinja_template_wrapper.py && \
cp /go/pkg/mod/github.com/llm-d/llm-d-kv-cache-manager@${KVCACHE_MANAGER_VERSION}/pkg/preprocessing/chat_completions/requirements.txt \
cp /go/pkg/mod/${KVCACHE_MANAGER_PATH}@${KVCACHE_MANAGER_VERSION}/pkg/preprocessing/chat_completions/requirements.txt \
/workspace/kv-cache/requirements.txt

# HuggingFace tokenizer bindings (static lib)
RUN mkdir -p lib
# Ensure that the RELEASE_VERSION matches the one used in the imported llm-d-kv-cache-manager version
# Ensure that the RELEASE_VERSION matches the one used in the imported llm-d-kv-cache version
ARG RELEASE_VERSION=v1.22.1
RUN curl -L https://github.com/daulet/tokenizers/releases/download/${RELEASE_VERSION}/libtokenizers.${TARGETOS}-${TARGETARCH}.tar.gz | tar -xz -C lib
RUN ranlib lib/*.a
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ test-unit: test-unit-epp test-unit-sidecar ## Run unit tests
.PHONY: test-unit-%
test-unit-%: download-tokenizer install-python-deps check-dependencies ## Run unit tests
@printf "\033[33;1m==== Running Unit Tests ====\033[0m\n"
@KV_CACHE_PKG=$$(go list -m -f '{{.Dir}}/pkg/preprocessing/chat_completions' github.com/llm-d/llm-d-kv-cache-manager 2>/dev/null || echo ""); \
@KV_CACHE_PKG=$$(go list -m -f '{{.Dir}}/pkg/preprocessing/chat_completions' github.com/llm-d/llm-d-kv-cache 2>/dev/null || echo ""); \
PYTHONPATH="$$KV_CACHE_PKG:$(VENV_DIR)/lib/python$(PYTHON_VERSION)/site-packages" \
CGO_CFLAGS=${$*_CGO_CFLAGS} CGO_LDFLAGS=${$*_CGO_LDFLAGS} go test $($*_LDFLAGS) -v $$($($*_TEST_FILES) | tr '\n' ' ')

Expand Down
1 change: 1 addition & 0 deletions deploy/config/sim-epp-kvcache-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ plugins:
blockSize: 16 # must match vLLM block size if not default (16)
hashSeed: "42" # must match PYTHONHASHSEED in vLLM pods
tokenizersPoolConfig:
modelName: <model-name> # specify the model name to use for tokenizer loading
hf:
tokenizersCacheDir: "/cache/tokenizers"
kvBlockIndexConfig:
Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,7 @@ plugins:
blockSize: 64
hashSeed: "12345"
tokenizersPoolConfig:
modelName: hf-repo/model-name
hf:
huggingFaceToken: your_hf_token_here # automatically set by `HF_TOKEN` environment variable
kvBlockIndexConfig:
Expand Down
8 changes: 5 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ require (
github.com/google/uuid v1.6.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/jellydator/ttlcache/v3 v3.4.0
github.com/llm-d/llm-d-kv-cache-manager v0.4.0
github.com/onsi/ginkgo/v2 v2.27.4
github.com/onsi/gomega v1.39.0
github.com/llm-d/llm-d-kv-cache v0.4.0
github.com/onsi/ginkgo/v2 v2.27.3
github.com/onsi/gomega v1.38.3
github.com/openai/openai-go v1.12.0
github.com/prometheus/client_golang v1.23.2
github.com/stretchr/testify v1.11.1
Expand Down Expand Up @@ -138,3 +138,5 @@ require (
sigs.k8s.io/structured-merge-diff/v6 v6.3.1 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
)

replace github.com/llm-d/llm-d-kv-cache => github.com/sagearc/llm-d-kv-cache-manager v0.0.0-20260105163019-e73ed03ca6ff
Comment thread
sagearc marked this conversation as resolved.
Outdated
12 changes: 6 additions & 6 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,6 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/llm-d/llm-d-kv-cache-manager v0.4.0 h1:MBWVpDW0PWsqNJEEAW1esrJW+Xavb0a7w14tCJWWyRY=
github.com/llm-d/llm-d-kv-cache-manager v0.4.0/go.mod h1:ZlK7MCuz5D/weLeHyNKEmVF/eJZDyYn3XyRowTihq9o=
github.com/mailru/easyjson v0.9.0 h1:PrnmzHw7262yW8sTBwxi1PdJA3Iw/EKBa8psRf7d9a4=
github.com/mailru/easyjson v0.9.0/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo=
Expand All @@ -210,10 +208,10 @@ github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid/v2 v2.1.1 h1:suPZ4ARWLOJLegGFiZZ1dFAkqzhMjL3J1TzI+5wHz8s=
github.com/oklog/ulid/v2 v2.1.1/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ=
github.com/onsi/ginkgo/v2 v2.27.4 h1:fcEcQW/A++6aZAZQNUmNjvA9PSOzefMJBerHJ4t8v8Y=
github.com/onsi/ginkgo/v2 v2.27.4/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
github.com/onsi/gomega v1.39.0 h1:y2ROC3hKFmQZJNFeGAMeHZKkjBL65mIZcvrLQBF9k6Q=
github.com/onsi/gomega v1.39.0/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
github.com/onsi/ginkgo/v2 v2.27.3 h1:ICsZJ8JoYafeXFFlFAG75a7CxMsJHwgKwtO+82SE9L8=
github.com/onsi/ginkgo/v2 v2.27.3/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
github.com/onsi/gomega v1.38.3 h1:eTX+W6dobAYfFeGC2PV6RwXRu/MyT+cQguijutvkpSM=
github.com/onsi/gomega v1.38.3/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
github.com/openai/openai-go v1.12.0 h1:NBQCnXzqOTv5wsgNC36PrFEiskGfO5wccfCWDo9S1U0=
github.com/openai/openai-go v1.12.0/go.mod h1:g461MYGXEXBVdV5SaR/5tNzNbSfwTBBefwc+LlDCK0Y=
github.com/pebbe/zmq4 v1.4.0 h1:gO5P92Ayl8GXpPZdYcD62Cwbq0slSBVVQRIXwGSJ6eQ=
Expand Down Expand Up @@ -248,6 +246,8 @@ github.com/redis/go-redis/v9 v9.11.0/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagearc/llm-d-kv-cache-manager v0.0.0-20260105163019-e73ed03ca6ff h1:EuIc0hACNvojkDpJUywFnnzMC2wZG2pllsHshr/5Xi4=
github.com/sagearc/llm-d-kv-cache-manager v0.0.0-20260105163019-e73ed03ca6ff/go.mod h1:ZGFOrk5hsx0Cov5bQaU+u5JgKgzgbLe/NfQ6ZNquYmE=
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
Expand Down
13 changes: 8 additions & 5 deletions pkg/plugins/scorer/precise_prefix_cache.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ import (
"fmt"
"os"

"github.com/llm-d/llm-d-kv-cache-manager/pkg/kvcache"
"github.com/llm-d/llm-d-kv-cache-manager/pkg/kvcache/kvevents"
preprocessing "github.com/llm-d/llm-d-kv-cache-manager/pkg/preprocessing/chat_completions"
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache"
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvblock"
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvevents"
preprocessing "github.com/llm-d/llm-d-kv-cache/pkg/preprocessing/chat_completions"
"sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
Expand Down Expand Up @@ -85,16 +86,18 @@ func PrecisePrefixCachePluginFactory(name string, rawParameters json.RawMessage,
// If the configuration is invalid or if the indexer fails to initialize,
// an error is returned.
func New(ctx context.Context, config PrecisePrefixCachePluginConfig) (*PrecisePrefixCacheScorer, error) {
tokenProcessor := kvblock.NewChunkedTokenDatabase(kvblock.DefaultTokenProcessorConfig())

// initialize the indexer
kvCacheIndexer, err := kvcache.NewKVCacheIndexer(ctx, config.IndexerConfig)
kvCacheIndexer, err := kvcache.NewKVCacheIndexer(ctx, config.IndexerConfig, tokenProcessor)
if err != nil {
return nil, fmt.Errorf("failed to create `kvcache.Indexer`: %w", err)
}

go kvCacheIndexer.Run(ctx)

// initialize the KV-events pool
pool := kvevents.NewPool(config.KVEventsConfig, kvCacheIndexer.KVBlockIndex())
pool := kvevents.NewPool(config.KVEventsConfig, kvCacheIndexer.KVBlockIndex(), tokenProcessor)
pool.Start(ctx)

return &PrecisePrefixCacheScorer{
Expand Down
10 changes: 5 additions & 5 deletions pkg/plugins/scorer/precise_prefix_cache_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ import (
"testing"

"github.com/google/go-cmp/cmp"
"github.com/llm-d/llm-d-kv-cache-manager/pkg/kvcache"
"github.com/llm-d/llm-d-kv-cache-manager/pkg/kvcache/kvblock"
"github.com/llm-d/llm-d-kv-cache-manager/pkg/kvcache/kvevents"
preprocessing "github.com/llm-d/llm-d-kv-cache-manager/pkg/preprocessing/chat_completions"
"github.com/llm-d/llm-d-kv-cache-manager/pkg/tokenization"
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache"
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvblock"
"github.com/llm-d/llm-d-kv-cache/pkg/kvcache/kvevents"
preprocessing "github.com/llm-d/llm-d-kv-cache/pkg/preprocessing/chat_completions"
"github.com/llm-d/llm-d-kv-cache/pkg/tokenization"
"github.com/stretchr/testify/require"
k8stypes "k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
Expand Down
12 changes: 6 additions & 6 deletions scripts/fetch-python-wrapper.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,32 +16,32 @@ set -euo pipefail
VERSION="${1:-${KVCACHE_MANAGER_VERSION:-v0.3.2}}"
OUTPUT_DIR="${2:-llm-d-kv-cache-manager/pkg/preprocessing/chat_completions}"

@hyeongyun0916 hyeongyun0916 Jan 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we should also remove -manager here.


REPO_URL="https://github.com/llm-d/llm-d-kv-cache-manager.git"
REPO_URL="https://github.com/llm-d/llm-d-kv-cache.git"
WRAPPER_FILE="pkg/preprocessing/chat_completions/render_jinja_template_wrapper.py"

# Create temporary directory (equivalent to: mkdir -p /tmp/kv-cache-manager)
# TEMP_DIR will be an absolute path like /tmp/tmp.XXXXXX
TEMP_DIR=$(mktemp -d)
trap "rm -rf ${TEMP_DIR}" EXIT

echo "Fetching Python wrapper from llm-d-kv-cache-manager@${VERSION}..."
echo "Fetching Python wrapper from llm-d-kv-cache@${VERSION}..."

# Equivalent to: cd /tmp/kv-cache-manager && git clone ... .
# Equivalent to: cd /tmp/kv-cache && git clone ... .
# (clones repo contents directly into TEMP_DIR - using absolute path, no need to cd)
git clone --depth 1 --branch "${VERSION}" "${REPO_URL}" "${TEMP_DIR}"

# Create output directory if it doesn't exist
# (equivalent to: mkdir -p /workspace/llm-d-kv-cache-manager/pkg/preprocessing/chat_completions)
# (equivalent to: mkdir -p /workspace/llm-d-kv-cache/pkg/preprocessing/chat_completions)
# OUTPUT_DIR is relative to current working directory (relative path, same as original)
mkdir -p "${OUTPUT_DIR}"

# Copy wrapper file
# Source: absolute path ${TEMP_DIR}/${WRAPPER_FILE} (e.g., /tmp/tmp.XXXXXX/pkg/.../wrapper.py)
# Destination: relative path ${OUTPUT_DIR}/ (e.g., llm-d-kv-cache-manager/pkg/.../)
# Destination: relative path ${OUTPUT_DIR}/ (e.g., llm-d-kv-cache/pkg/.../)
# (equivalent to original: cp pkg/.../wrapper.py /workspace/... from within temp dir)
cp "${TEMP_DIR}/${WRAPPER_FILE}" "${OUTPUT_DIR}/"

# Cleanup happens automatically via trap (equivalent to: rm -rf /tmp/kv-cache-manager)
# Cleanup happens automatically via trap (equivalent to: rm -rf /tmp/kv-cache)

echo "Successfully fetched render_jinja_template_wrapper.py to ${OUTPUT_DIR}/"

2 changes: 1 addition & 1 deletion scripts/kubernetes-dev-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ if [[ -z "${HF_TOKEN:-}" ]]; then
exit 1
fi

export VLLM_CHART_DIR="${VLLM_CHART_DIR:-../llm-d-kv-cache-manager/vllm-setup-helm}"
export VLLM_CHART_DIR="${VLLM_CHART_DIR:-../llm-d-kv-cache/vllm-setup-helm}"
# Check that Chart.yaml exists
if [[ ! -f "$VLLM_CHART_DIR/Chart.yaml" ]]; then
echo "Chart.yaml not found in $VLLM_CHART_DIR"
Expand Down
1 change: 1 addition & 0 deletions test/e2e/e2e_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,7 @@ plugins:
blockSize: 16 # must match vLLM block size if not default (16)
hashSeed: "42" # must match PYTHONHASHSEED in vLLM pods
tokenizersPoolConfig:
modelName: Qwen/Qwen2.5-1.5B-Instruct
hf:
tokenizersCacheDir: "/cache/tokenizers"
kvBlockIndexConfig:
Expand Down