[avlis]

Summary

Adds Debian packaging producing two packages with dpkg-buildpackage -b:

• gufo — the binary, runtime data, license notices, a dedicated gufo
  system user (primary group llm-servers, plus video/render), a classic
  gufo.service and an instantiated gufo@.service, the shared launcher
  /usr/lib/gufo/gufo-launch.sh, and admin configuration through
  /etc/gufo/gufo.env plus /etc/gufo/instance.d/<name>.env.
• gufo-rocm-repo — registers the AMD ROCm apt repository (pinned to the
  7.2.x line) and its signing key, because hipblaslt, rocwmma and the HIP
  runtime Gufo links against are not in Debian or Ubuntu.

Plus deployment tooling (tools/deploy/) and a README "Debian package" section.

Configuration model

• /etc/gufo/gufo.env — host-wide facts, a dpkg conffile: the model library
  path, the cache root.
• /etc/gufo/instance.d/<name>.env — a serving configuration: model, draft
  sidecar, context, sessions, speculative backend, port, cache budget, log level.
  Admin-owned, never touched by the package.
• gufo@<name>.service + gufo-launch.sh — the launcher sources the host file,
  then the instance file, then builds the command line. The unit carries no
  model-, port- or cache-specific value, so a configuration change never means
  editing a unit and never needs a drop-in. The README's MTP/context recipe uses
  this route rather than an ExecStart override, which would be a second source
  of truth that silently wins on every start.
• Cache is private per instance (/var/lib/gufo/cache/instances/<name>,
  created as the service user) because the server rejects a cache directory it
  does not own; weights are shared in /var/lib/huggingface/hub, setgid
  llm-servers, so several servers on one machine reuse the same downloads.
  Both survive apt purge.

Service lifecycle

dh_installsystemd --no-enable --no-start emits neither a start nor a stop, so
both sides are explicit.

Instances cannot be enumerated the obvious way: deb-systemd-invoke passes unit
names verbatim and does not expand gufo@.service; systemctl is-enabled gufo@.service answers indirect; systemctl list-unit-files 'gufo*' does not
list an enabled instance at all, because enabling an instance is a wants symlink
named after the instance. prerm therefore expands the wants symlinks in
/etc/systemd/system and /run/systemd/system for gufo.service and
gufo@*.service, unions that with systemctl list-units --all --type=service 'gufo*' to catch units started without being enabled, drops the bare template
(systemd refuses to act on it), marks each live unit in
/run/gufo/<unit>.was-active — one marker per unit, a host may run several —
stops each, and prints a note to stderr when it found nothing running.

postinst then decides, in /usr/lib/gufo/gufo-upgrade-units.sh, called with
dpkg's configured-from version ($2: empty on first install, the previous
version on upgrade or reinstall):

unit state at configure	action	because
active	restart	its binary was replaced underneath it; start is a no-op
inactive, marker stamped with this boot	restart	prerm stopped it in this transaction; restart on an inactive unit starts it
inactive, enabled, rebooted since prerm	start	/run went with the markers; booting is the recovery the admin asked for
inactive, enabled, no reboot since prerm	nothing	enablement describes boot, not now: the unit was stopped deliberately
inactive, marker from an earlier boot	nothing	a marker belongs to its own transaction; one left by a crashed upgrade must not resurrect a unit
inactive, not enabled	nothing	the admin's choice, never overridden
bare gufo@.service	filtered out	systemd refuses to act on the template

Why the boot id. Enablement is a statement about boot, not about the present moment, so
is-enabled cannot distinguish "the machine rebooted, /run was wiped with the markers, and
the admin wants this up" from "an operator stopped it on a live host to keep the device
free" — and getting that wrong takes a device and re-enters a load balancer's path. prerm
therefore writes the boot id twice: inside each marker, and in /var/lib/gufo/upgrade-boot.id
outside /run, which is the only thing that survives the reboot the distinction depends on.
start fires only when the recorded id differs from /proc/sys/kernel/random/boot_id, and
configure consumes that record, so a reboot predating an upgrade can never start anything.
A marker is honoured only when it names the running kernel's boot id, and prerm clears
markers before writing its own: a transaction that dies between prerm and configure leaves
no live claim on the next one.

One line of evidence is printed per unit — restart, start or leave, each with the
reason above; a failed action exits non-zero so
postinst warns instead of failing the install silently.

Why the decision is not in prerm: dpkg runs the prerm of the package
being upgraded from. A prerm-only rule therefore cannot govern the upgrade that
first introduces it, and its start fallback is a no-op on a unit that is still
active — which leaves the unit executing the inode the package replaced while
dpkg -l and gufo --version report the new one. postinst is this package's own
script, so the rule holds from the first upgrade that carries it. prerm's markers
remain, as the record of what was running before it was stopped.

transition	result
first install	stays stopped; admin configures /etc/gufo/instance.d/<name>.env, then systemctl enable --now gufo@<name>
same-version reinstall while running	stopped, unpacked, back active with a new pid, no manual step
upgrade while running	every active unit restarted; /proc/<pid>/exe → /usr/bin/gufo
upgrade with the unit deliberately stopped and not enabled	stays stopped
remove / purge	unit stopped, markers and /run/gufo removed, user and group dropped, shared cache and home kept

A restart is model-load downtime. Measured on gfx1151 at 128k context with two
sessions: ~25 s with the weights in the page cache, ~96 s cold — the spread is the
page cache, not the release, so budget for the cold figure on a host that has not run
this model recently. Upgrades therefore belong in a window with a second backend in front,
and the backend health check should be readiness-based (GET /ready, which is 503 until a
model is resident) rather than liveness-based, so a reloading host withdraws itself.

Source revision stamp

The binary advertises its revision through -DGUFO_REVISION, computed by
debian/get-revision.sh, which writes nothing to the tree:

tree state	stamp
packager sets GUFO_REVISION=…	used verbatim — the packager owns provenance
clean checkout at a tag	X.Y.Z
clean checkout N commits past a tag	X.Y.Z-N-g<sha>
tracked sources modified	-dirty
debian/changelog regenerated by this packaging	not -dirty — that path alone is excluded
source tarball with no .git	unknown

--tags is required because the release tags here are lightweight, and
--git-dir/--work-tree are pinned to the source root so a tarball extracted
inside another repository cannot inherit that repository's HEAD. Because the stamp is
read at build time, docs/RELEASING.md states the order of operations: commit,
then build, then verify the stamp from the artifact before deploying.

Key decisions

• Runtime-only installs. ROCm's runtime packages recommend their -dev
  counterparts, which pull the compiler and the GCC toolchain; every install path
  uses --no-install-recommends.
• ROCm loader path. AMD installs into /opt/rocm/lib without registering it,
  so libhipblas/librocblas/libamdhip64 fail to load. The package ships
  /etc/ld.so.conf.d/rocm.conf and runs ldconfig in postinst.
• Explicit ROCm dependencies in control: AMD's debs ship no shlibs
  metadata, so dpkg-shlibdeps cannot derive them.
• Crash-loop guard. StartLimitIntervalSec/StartLimitBurst are [Unit]
  keys — in [Service] systemd ignores them silently — and a crash loop must not
  reload a ~100 GiB model repeatedly.
• Unit filesystem hardening, scoped to what the workload proves it needs:
  ProtectSystem=full, ProtectHome=yes, PrivateTmp=yes and one explicit
  ReadWritePaths=/var/lib/gufo on both units. The set was derived by measuring what the
  running instance holds open — file descriptors under the model cache (read-only mmaps)
  and /var/lib/gufo/cache/instances/<instance> (the disk cache the launcher mkdirs) —
  plus the shipped env files (no /home path) and the launcher (no /tmp, no writes under
  /usr). Deliberately not set: ProtectSystem=strict, ProtectKernelTunables and
  NoNewPrivileges. Some sources read /sys for device information and the ROCm kfd/dri
  access path is unproven under a read-only /sys or a no-new-privileges domain; the failure
  mode of those three is a unit that will not start, which costs a full model reload — a
  larger outage than the exposure they remove. Each should be widened only with a cold start
  and a real completion measured under it, and the reason is written into both unit files.
  Verified in a namespace, not just declared: PrivateTmp resolves to a
  systemd-private-… mount, touch /usr/bin/… fails Read-only file system, /home is
  empty, and the cache path keeps a read-write descriptor.
• install-gufo-deb.sh resolves the newest package with sort -V | tail -1,
  so a new release needs no script change and a plain ls | head -1 cannot
  re-install an older one that sorts first.

Contents

debian/ — two packages, gufo.service, gufo@.service, gufo-launch.sh,
maintainer scripts, gufo-upgrade-units.sh (shipped to /usr/lib/gufo/, and the
part of the lifecycle that can be tested with a stubbed service manager), get-revision.sh,
gen-changelog.sh, rules, tests/ (the upgrade-path matrix and the container lifecycle test,
both executable, described under Verified) plus the release-workflow hook that generates the
changelog entry; tools/deploy/install-gufo-deb.sh and
tools/deploy/sync-model-cache.sh; the README "Debian package" section;
docs/RELEASING.md.

Builds without a wrapper

The package is the interface. dpkg-buildpackage -b -us -uc must work on a machine nobody
has ssh'd into, with no helper script, no exported flags and no vendor-specific prefix:

• Build-Depends is complete. find_package(hip) needs hip-dev, and .hip device
  code needs ROCm's clang from rocm-llvm — gcc has no amdgcn target. python3 is
  resolved by CMake directly. Every one of those is declared, so a clean chroot resolves
  them itself rather than succeeding only on a machine that happened to have them.
• The ROCm prefix is discovered, not assumed: hipconfig's HIP_PATH, overridable with
  DEB_ROCM_PREFIX for a staging chroot, a ROCm under $HOME, or a multi-version host.
  /opt/rocm is only the last fallback, and a wrong value fails loudly at
  find_package(hip) instead of being silently right on one maintainer's box.
• Machine tuning is not baked in. No -march= and no pinned compiler appear anywhere in
  debian/: a packager's CPU flags belong to the packager, and the build must be correct on
  hardware its author does not own.
• No network, no root, no host contact at build time; Rules-Requires-Root: no,
  no curl/wget/apt in any maintainer script. Source tarballs without .git are
  handled by debian/get-revision.sh.

Verified

• Clean-room install in a bare debian:trixie container: ldd /usr/bin/gufo resolves,
  apt purge stops the unit and removes the package files while keeping the shared cache
  and home.
• Inference inside a container is not covered by the shipped test. The container test
  replaces ExecStart with a stub, so no weights are loaded and no throughput is claimed;
  gufo diagnose does run there and only enumerates the device. Serving a model inside a
  container is a manual, GPU-passthrough bring-up documented in the packaging notes
  (2026-10-10), where gufo.service reached active (running) and answered /v1/models
  and a chat completion — it is not part of debian/tests/, and CI should not be expected
  to reproduce it on a host with a resident model.
• Runtime-only install adds no compiler packages or executables.
• Live, gfx1151, Flash-Next Q4 + MTP at 128k context behind a load balancer:
  package-driven upgrades restart the instance unattended, printing
  gufo: restart gufo@<name>.service …, after which readlink /proc/<pid>/exe
  is /usr/bin/gufo and not (deleted), the marker directory is empty, and
  front-end probes stay 200 across the reload window on the second backend.
• check-docs.py passes; no C++ or Python files are touched.
• debian/tests/upgrade-units: 19 scenarios over the lifecycle table above, runnable as
  make test or debian/tests/upgrade-units; systemctl and deb-systemd-invoke are stubbed,
  so it exercises the decision without a service manager.
• debian/tests/test_deb_package_in_container.sh: the same lifecycle against real dpkg and
  real systemd in debian:trixie - prerm before unpack, postinst after, marker files as the
  package writes them. 34 assertions, 0 failed on the artifact under review, and 33 assertions,
  0 failed replaying an upgrade from 0.15.1-1 — a release whose prerm wrote empty
  markers, so the postinst falls back to the state record (it prints
  restart gufo@legacy.service · marked by an older prerm; boot provenance from the state record). Scenario E also runs self-contained, patching the installed prerm to emit
  empty markers, so the replay needs no historical binary if none is kept. Takes the
  .deb(s) as arguments; image, port, model cache and device passthrough are flags or
  environment, and GPU assertions are skipped with a printed reason where there is no device.
  ExecStart is replaced by a stub inside those containers, so this covers packaging and service
  lifecycle, not inference. It is deliberately not listed in debian/tests/control: it needs
  a privileged container, which autopkgtest would fail for reasons unrelated to the packaging.

Accepted lintian findings

package-modifies-ld.so-search-path (intentional, see above);
maintscript-calls-ldconfig / command-with-path-in-maintainer-script;
custom-library-search-path RUNPATH /usr/lib/llvm-19/lib (libomp);
hardening-no-pie (the HIP toolchain emits no PIE objects, consistent with the
Nix build); no-manual-page (the project ships none).

Notes

• The apt pin is one Package: line of space-separated globs, not one line per glob. apt reads
  a single Package: field per preferences stanza, so a stanza carrying six of them installs without
  complaint, sits in /etc/apt/preferences.d looking configured, and never applies the priority it
  asks for — the distribution's 500 stays in force and nothing reports it. debian/tests/ now
  asserts both halves: that the installed stanza has exactly one Package: field (no network needed)
  and that apt-cache policy resolves a ROCm package to 600 (reported as a skip when the sandbox
  fetched no index, never as a pass it did not earn).
• The ROCm patch version is pinned in gufo-rocm-repo.postinst; bump per release.
  The package tracks the qualified toolchain line in the README and takes the
  newest patch of it, so the deb stays ABI-compatible.
• Do not apt remove ROCm -dev packages on hosts with xrt/dkms — AMD's
  dependency graph cascades into unrelated packages.
• Over ssh always install with -o Dpkg::Options::=--force-confold: an
  interactive conffile prompt hits EOF and strands the package half-configured
  (iU).
• Follow-ups worth considering: CI build of the .deb on release; the changelog
  trailer format in generated entries; a CI lane that can load a small model in a
  privileged container, which is the one coverage debian/tests/ still lacks (see
  Verified), and readiness-based backend health checks where a balancer fronts the package.

[iwr-redmond]

What is the reasoning behind choosing ROCm 7.2.4 rather than the newer 10.0?

Answer: it is the latest.

[avlis]

The package tracks the project's qualified toolchain. README pins that at ROCm 7.2.3 (GCC 15.3); 7.2.4 is the latest patch of that line from AMD's repository, so the deb stays ABI-compatible with the qualified version while picking up the newest patch. ROCm 10 is not qualified for gfx1151 yet — there's an open draft PR (#228) evaluating it, and shipping production packages on an unqualified math-library toolchain would bypass the project's quality checks. The ROCm version is pinned in one place (gufo-rocm-repo.postinst) specifically so it can be bumped to 10.x per release once qualified.

[fedeizzo]

We are currently working on the stability of gufo (tool calling, caching, etc.). We'll take a look in the new few days, hopefully. thanks for the pr

[avlis]

We are currently working on the stability of gufo (tool calling, caching, etc.). We'll take a look in the new few days, hopefully. thanks for the pr

no stress, no hurry. just something that I needed for myself, and I wanted to give something back to your incredible feat. keep up the good work! :o)

[avlis]

actually, on my own testing on my bosgame M5 with 128GB of ram, I was having some bad TTFT, and it seems to be related to disk trashing on the n-gram table: the Flash-Next PLE n-gram table is disk-resident by design — it's a 28.8 GB table (320M rows, IQ4_NL) read straight from the GGUF with O_DIRECT, and the row cache was a hardcoded 8 MB. Every prefill token does 16 row lookups, so a cache-miss prefill of 70–95k tokens means ~1.5M random 4KB reads (~5–6 GB) per request.
My bosgame saturated the NVMe at 50 MB/s effective: TTFT jumped from the 70 s compute floor to 88–120 s, and the I/O pressure (30–40% PSI) dragged concurrent decode down too. Short prompts were fine (the 8 MB cache absorbs them) — it only showed up with long agent-style prefills.

My fix: made the row cache size configurable with --ngram-cache-bytes (default 512 MiB). At 1 GiB the cache holds ~11.8M rows — enough to cover every distinct row a long prefill touches — so the disk reads collapse. Measured: prefill went from 645–1189 tok/s (I/O-bound) to 1425 tok/s (compute-bound, matching the benchmark), a 70k-token prefill's TTFT dropped to ~49 s, disk reads went from 508 MB/10 s to 0, and cache-hit turns are back to 50 ms TTFT. The ngram table design (O_DIRECT, avoiding page-cache pollution) stays intact — we just sized the cache for the workload instead of the benchmark.

can submit it as a new PR if you want, but it's simple enough:

// ngram.cpp — the actual change
const std::size_t cache_bytes_resolved =
cache_bytes == 0 ? NgramTable::kDefaultCacheBytes : cache_bytes;
t->cache_count_ = std::bit_floor(cache_bytes_resolved / t->row_bytes_);
// ngram.hpp
static constexpr std::size_t kDefaultCacheBytes = 512 * 1024 * 1024;
// serve.cpp — run path only (the help path is identical)
llm_parser.AddOption("", "--ngram-cache-bytes", "N",
"RAM row cache for the Flash-Next PLE n-gram table (default: "
"536870912 = 512 MiB; ...)", "Cache", &ngram_cache_bytes);
...
backend->load(..., TextRunnerRamCacheOptions{...}, ngram_cache_bytes)

[iwr-redmond]

Adding a CI build for releases is going to make Gufo much easier for end users to adopt. Nice!

[fedeizzo]

@avlis We would like to keep the scaffolding for packaging/etc. outside of this repo, and collect everything in https://github.com/gufo-org/toolboxes, do you moving everything there?

Sorry for the long delay, but we wanted to focus on stability at first. We are now trying to reduce the backlog of PRs/issues

[avlis]

Reply to @fedeizzo — packaging scaffolding to gufo-org/toolboxes

Agreed, and I think the move is right for a reason stronger than preference: a host
.deb and a Nix/OCI image are not the same artifact and do not deserve the same care
from the same repo. The .deb lane carries dpkg lifecycle, conffile policy and an apt
pin — machinery the engine source has no reason to hold.

What we propose

A single PR to toolboxes, adding a deb/ lane as a sibling of nix/ and systemd/,
in the shape docker-ce-packaging uses for the same problem (packaging for a source repo
it does not live in):

toolboxes/
├── deb/
│   ├── common/            # control rules postinst postrm prerm gufo.service gufo@.service
│   │   └── source/format  # gufo.env gufo-launch.sh rocm.conf rocm.gpg upgrade-units.sh
│   ├── build-deb          # clones gufo at a rev, assembles debian/ into it, dpkg-buildpackage
│   ├── gen-deb-ver       # version + revision + `~` rules; takes the SOURCE dir as $1
│   ├── gen-changelog.sh
│   ├── debian-trixie.Dockerfile   # build env, and the real-dpkg/real-systemd test env
│   └── tests/            # the 19-case upgrade matrix + the container lifecycle test
└── .github/workflows/build_and_publish_deb.yml

No new automation is needed in gufo to drive it: release-please.yml already sends a
gufo-release repository dispatch (#309) and your publish_release.yml already consumes
that event type, so the deb lane hangs off the seam you built. If you would rather the
flake produce the .deb, say so and we will write it that way instead — the tree is the
same either way.

Facts that must travel with the tree

• The apt pin. A stanza with six Package: lines does not scope six packages; it is
  not the documented form and resolves to priority 500. The fix in this PR is one glob
  line (Package: hip* roc* rocm* rdc* *roctracer* *hsa*) and it is the difference
  between a host that takes a third-party ROCm build back from the distro and one that
  silently does not. Proven in a sandbox: apt-cache policy reports 600 for the fixed
  stanza and 500 for the six-line one, same key, same suite.
• The lifecycle matrix. 19 upgrade cases — prerm from a pre-fix version, prerm-marked,
  first install, admin-enabled, disabled-and-stopped (must never start), /run wiped,
  classic unit plus instances together, bare template (must never be acted on), failing
  restart (must exit non-zero and warn), --dry-run (mutates nothing) — plus the same
  suite against real dpkg and real systemd in a container.
• ROCm stays qualified. The qualified toolchain in the README is GCC 15.3 and ROCm
  7.2.3; the package floors its dependency at 7.2.53211.70203 and takes the newest
  patch of that line. The ROCm 10.1 work stays out of this lane until it is qualified,
  and it says so in its own metadata rather than in a comment.
• One asymmetry we will document instead of hide. Your Nix package applies
  rocr-fixed-cwsr.patch; a host install gets AMD's apt ROCm, which does not carry it.
  So a .deb install is not equivalent to your qualified build. We will state that in
  the packaging docs rather than let someone discover it under load. If it should be
  equivalent, the fix is a patched ROCr in your lane — not a patch we can ship from ours.

Three answers that decide the shape

1. Does deb/ live in toolboxes with its own workflow triggered by the existing
   gufo-release dispatch — or should the flake produce the .deb?
2. Where do the built packages go: release assets, OCI artifacts on ghcr, or built but
   not published? We would rather not stand up a signed apt feed unless someone wants its
   key, its suite migrations and its retention policy.
3. After the move, is debian/changelog generated in the packaging repo from the release
   tag? If yes, our changelog-sync step in release-please.yml is dead weight and this PR
   stops touching the engine's release automation entirely — which is the cleanest end
   state I can see for this thread.

This PR stays open until the toolboxes PR exists, so the review history here stays
reachable, and then closes with a pointer to it.

[fedeizzo]

Thanks for the detailed proposal. Moving the packaging into Toolboxes sounds good. Please open a PR there, then close this one with a link so we retain the discussion.

For that first PR, I’d like to keep the scope as small as possible: a native Debian build that packages the executable, required runtime files and licenses, with the dependencies needed to run it declared correctly.

Please leave systemd services, launchers, environment-based configuration, model/cache management, deployment helpers and custom service upgrade handling for follow-up PRs. Those introduce deployment choices that we should evaluate separately. The first goal is to validate the build, installation and release pipeline; then we can iterate based on what users actually need.

To answer your three questions:

1. Put the native Debian packaging and its workflow in Toolboxes, using the existing gufo-release dispatch. Let’s defer Nix-based Debian packaging.
2. Ideally, upload the packages as assets on the corresponding Gufo release. Cross-repository uploads are possible with the appropriate credentials; we can arrange those on our side.
3. Keep Gufo’s existing CHANGELOG.md as the release history. Generate only the minimal Debian changelog metadata in Toolboxes from the selected release tag, without changing Gufo’s release automation.

Please also keep the runtime distinction explicit: the AMD apt runtime does not include our ROCr patch, so this package should not be described as equivalent to the qualified Nix build.

A small first PR will be easier to review and validate. We can add service integration and other conveniences incrementally once the basic package pipeline works.

[avlis]

Closing this: the packaging has moved, as asked in review.

gufo-org/toolboxes now carries it as a deb/ lane beside nix/ and systemd/ —
gufo-org/toolboxes#17 — one commit, 32 files, +2943.

What travels unchanged: control, rules, the maintainer scripts, the units and gufo.env, the
upgrade-unit decision logic and both test suites, including the container test that runs the
lifecycle against real dpkg and real systemd with the .deb as its argument.

What changes is where they are built and how they are parameterised: files are assembled into
<checkout>/debian/ at build time rather than committed into the engine tree, the version comes
from gen-deb-ver reading version.txt plus the tag, and every machine-specific choice is a
default a clone overrides by environment instead of a local edit — CPU_FLAGS ships empty, so the
portable baseline is the default and tuning a fleet is something you opt into.

Two things from this thread are worth restating where they will be read:

• The apt pin is one Package: field of space-separated globs. Six Package: lines in one
  stanza install without complaint, sit in /etc/apt/preferences.d looking configured, and never
  apply the priority they ask for. The pin gate asserts both halves — the shape, and a ROCm family
  resolving at 600 — and carries a negative control that writes the broken shape and requires apt to
  report it unpinned in the same run, so the gate fails rather than silently becoming decoration.
• A maintainer-script fix is dormant for one upgrade, because dpkg runs the prerm of the release
  you upgrade from. That is why the restart logic lives in postinst.

This PR stays as the review history of the packaging and its branch is frozen rather than renamed —
renaming a pull request's head ref closes the pull request, and the review that shaped these files is
on this thread. Thanks for the review that sent it to the right repository.