Skip to content

feat(inference): add dormant llama.cpp request guard artifact #8586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ericksoa merged 5 commits into from
Aug 8, 2026
Merged

feat(inference): add dormant llama.cpp request guard artifact #8586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
55d682a
feat(inference): add llama.cpp request guard
ericksoa Aug 7, 2026
09019af
test(inference): strengthen request guard contracts
ericksoa Aug 7, 2026
d63a2d9
fix(inference): harden llama.cpp request guard
ericksoa Aug 7, 2026
1ae2dd4
test(inference): cover llama.cpp guard validation
ericksoa Aug 7, 2026
eb95e3d
test(inference): keep annotation fixtures linear
ericksoa Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions managed-inference/images/llama-cpp/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
ARG CUDA_DEV_IMAGE
ARG CUDA_RUNTIME_IMAGE

FROM ${CUDA_DEV_IMAGE} AS build

Check warning on line 7 in managed-inference/images/llama-cpp/Dockerfile

View workflow job for this annotation

GitHub Actions / Build native llama.cpp server (amd64)

Default value for global ARG results in an empty or invalid base image name 

InvalidDefaultArgInFrom: Default value for ARG ${CUDA_DEV_IMAGE} results in empty or invalid base image name
More info: https://docs.docker.com/go/dockerfile/rule/invalid-default-arg-in-from/

Check warning on line 7 in managed-inference/images/llama-cpp/Dockerfile

View workflow job for this annotation

GitHub Actions / Build native llama.cpp server (arm64)

Default value for global ARG results in an empty or invalid base image name 

InvalidDefaultArgInFrom: Default value for ARG ${CUDA_DEV_IMAGE} results in empty or invalid base image name
More info: https://docs.docker.com/go/dockerfile/rule/invalid-default-arg-in-from/

ARG LLAMA_CPP_REVISION
ARG LLAMA_CPP_ARCHIVE_SHA256
Expand Down Expand Up @@ -32,6 +32,7 @@
curl=8.5.0-2ubuntu10.11 \
g++-14=14.2.0-4ubuntu2~24.04.1 \
gcc-14=14.2.0-4ubuntu2~24.04.1 \
golang-go=2:1.22~2build1 \
libcurl4-openssl-dev=8.5.0-2ubuntu10.11 \
libssl-dev=3.0.13-0ubuntu3.12 \
&& rm -rf /var/lib/apt/lists/*
Expand Down Expand Up @@ -80,7 +81,24 @@
&& find /opt/llama.cpp/licenses -type d -exec chmod 0555 '{}' + \
&& find /opt/llama.cpp/licenses -type f -exec chmod 0444 '{}' +

WORKDIR /src/nemoclaw-request-guard

COPY request-guard/go.mod request-guard/*.go ./

RUN go test ./... \
&& CGO_ENABLED=0 go build \
-trimpath \
-ldflags='-s -w -buildid=' \
-o /opt/llama.cpp/bin/nemoclaw-llama-cpp-request-guard . \
&& mkdir -p /opt/llama.cpp/licenses/go \
&& set -- /usr/share/doc/golang-[0-9]*-go/copyright \
&& test "$#" -eq 1 \
&& cp "$1" /opt/llama.cpp/licenses/go/copyright \
&& chmod 0555 /opt/llama.cpp/bin/nemoclaw-llama-cpp-request-guard \
&& chmod 0555 /opt/llama.cpp/licenses/go \
&& chmod 0444 /opt/llama.cpp/licenses/go/copyright

FROM ${CUDA_RUNTIME_IMAGE} AS runtime

Check warning on line 101 in managed-inference/images/llama-cpp/Dockerfile

View workflow job for this annotation

GitHub Actions / Build native llama.cpp server (amd64)

Default value for global ARG results in an empty or invalid base image name 

InvalidDefaultArgInFrom: Default value for ARG ${CUDA_RUNTIME_IMAGE} results in empty or invalid base image name
More info: https://docs.docker.com/go/dockerfile/rule/invalid-default-arg-in-from/

Check warning on line 101 in managed-inference/images/llama-cpp/Dockerfile

View workflow job for this annotation

GitHub Actions / Build native llama.cpp server (arm64)

Default value for global ARG results in an empty or invalid base image name 

InvalidDefaultArgInFrom: Default value for ARG ${CUDA_RUNTIME_IMAGE} results in empty or invalid base image name
More info: https://docs.docker.com/go/dockerfile/rule/invalid-default-arg-in-from/

ARG CUDA_DEV_IMAGE
ARG CUDA_RUNTIME_IMAGE
Expand Down Expand Up @@ -122,6 +140,7 @@
/usr/bin/sh

COPY --from=build --chmod=0555 /opt/llama.cpp/bin/llama-server /usr/local/bin/llama-server
COPY --from=build --chmod=0555 /opt/llama.cpp/bin/nemoclaw-llama-cpp-request-guard /usr/local/bin/nemoclaw-llama-cpp-request-guard
COPY --from=build --chmod=0555 /opt/llama.cpp/lib/ /opt/llama.cpp/lib/
COPY --from=build /opt/llama.cpp/licenses/ /usr/local/share/licenses/

Expand Down
5 changes: 5 additions & 0 deletions managed-inference/images/llama-cpp/image.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ kind: ServerImageBuild

metadata:
id: llama-cpp-server.v1
annotations:
nemoclaw.nvidia.com/request-guard-state: dormant

spec:
repository: ghcr.io/nvidia/nemoclaw/llama-cpp-server
Expand Down Expand Up @@ -111,6 +113,7 @@ spec:
curl: 8.5.0-2ubuntu10.11
g++-14: 14.2.0-4ubuntu2~24.04.1
gcc-14: 14.2.0-4ubuntu2~24.04.1
golang-go: 2:1.22~2build1
libcurl4-openssl-dev: 8.5.0-2ubuntu10.11
libssl-dev: 3.0.13-0ubuntu3.12
cmake:
Expand Down Expand Up @@ -138,6 +141,8 @@ spec:
requiredPaths:
- /opt/llama.cpp/lib/libggml-cuda.so
- /usr/local/bin/llama-server
- /usr/local/bin/nemoclaw-llama-cpp-request-guard
- /usr/local/share/licenses/go/copyright
- /usr/local/share/licenses/llama.cpp/AUTHORS
- /usr/local/share/licenses/llama.cpp/LICENSE
forbiddenPaths:
Expand Down
3 changes: 3 additions & 0 deletions managed-inference/images/llama-cpp/request-guard/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module github.com/NVIDIA/NemoClaw/managed-inference/images/llama-cpp/request-guard

go 1.22
Loading
Loading