Skip to content

test(cli): add command-coverage smoke gate #1086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
buremba merged 1 commit into from
May 26, 2026
Merged

test(cli): add command-coverage smoke gate #1086

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,52 @@
- name: SDK lifecycle e2e (apply + prune + real worker turn)
run: bash scripts/sdk-e2e.sh

# CLI command-coverage smoke — the hard gate that EVERY `lobu` command runs,
# not just the SDK lifecycle (sdk-e2e covers that). Boots one `lobu run`
# (embedded Postgres + a deterministic mock provider, no key) under an isolated
# HOME and walks the whole command surface, asserting each runs or fails
# gracefully. A failure here fails CI → the PR goes red. See scripts/cli-smoke.sh.
cli-smoke:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4

- uses: ./.github/actions/setup-submodule
with:
deploy-key: ${{ secrets.OWLETTO_WEB_DEPLOY_KEY }}

- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.5

# Node 22 — the worker uses isolated-vm (abi127 prebuild); the runner's
# default Node major would segfault the sandbox runtime.
- uses: actions/setup-node@v4
with:
node-version: "22"
Comment on lines +167 to +181
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify mutable action refs and missing persist-credentials in this workflow.
set -euo pipefail
file=".github/workflows/ci.yml"

echo "== uses refs not pinned to SHA =="
rg -n '^\s*-\s*uses:\s*[^@]+@v[0-9]+' "$file" || true

echo
echo "== checkout steps lacking persist-credentials: false =="
python - <<'PY'
import re, pathlib
p = pathlib.Path(".github/workflows/ci.yml")
lines = p.read_text().splitlines()
for i,l in enumerate(lines,1):
    if re.search(r'^\s*-\s*uses:\s*actions/checkout@', l):
        window = "\n".join(lines[i:i+8])
        if "persist-credentials: false" not in window:
            print(f"checkout without persist-credentials:false near line {i}")
PY

Repository: lobu-ai/lobu

Length of output: 1756

🏁 Script executed:

#!/bin/bash
set -euo pipefail
file=".github/workflows/ci.yml"

echo "== Context around lines 160-190 =="
nl -ba "$file" | sed -n '150,200p'

echo
echo "== Context around all actions/checkout steps =="
python - <<'PY'
import pathlib, re
p = pathlib.Path(".github/workflows/ci.yml")
lines = p.read_text().splitlines()
for i,l in enumerate(lines,1):
    if re.search(r'^\s*-\s*uses:\s*actions/checkout@', l):
        start=max(1,i-3); end=min(len(lines), i+12)
        print(f"\n--- checkout near line {i} (show {start}-{end}) ---")
        for ln in range(start,end+1):
            print(f"{ln:5d}: {lines[ln-1]}")
PY

Repository: lobu-ai/lobu

Length of output: 152

🏁 Script executed:

#!/bin/bash
set -euo pipefail
file=".github/workflows/ci.yml"

echo "== Context around lines 160-190 (cat -n) =="
cat -n "$file" | sed -n '150,200p'

echo
echo "== Exact checkout blocks (uses actions/checkout@) with nearby lines =="
python - <<'PY'
import pathlib, re
p = pathlib.Path(".github/workflows/ci.yml")
lines = p.read_text().splitlines()
for i,l in enumerate(lines,1):
    if re.search(r'^\s*-\s*uses:\s*actions/checkout@', l):
        start=max(1,i-2); end=min(len(lines), i+14)
        print(f"\n--- checkout near line {i} (show {start}-{end}) ---")
        for ln in range(start,end+1):
            print(f"{ln:5d}: {lines[ln-1]}")
PY

Repository: lobu-ai/lobu

Length of output: 11098

Pin CI actions by commit SHA and disable checkout credential persistence

In cli-smoke (steps around lines 167-181), actions/checkout@v4, oven-sh/setup-bun@v2, and actions/setup-node@v4 use mutable tags, and the actions/checkout step does not set persist-credentials: false, increasing token exposure risk. The same actions/checkout@v4 + missing persist-credentials: false pattern appears in multiple other jobs in ci.yml.

Suggested patch 
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@<pinned-sha>
+        with:
+          persist-credentials: false

-      - uses: oven-sh/setup-bun@v2
+      - uses: oven-sh/setup-bun@<pinned-sha>
         with:
           bun-version: 1.3.5

-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@<pinned-sha>
         with:
           node-version: "22"
🧰 Tools
🪛 zizmor (1.25.2)

[warning] 167-167: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

[error] 167-167: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

[error] 173-173: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

[error] 179-179: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 167 - 181, In the cli-smoke job,
replace mutable action tags with fixed commit SHAs for actions/checkout,
oven-sh/setup-bun and actions/setup-node and set actions/checkout to
persist-credentials: false; specifically update the step that references
actions/checkout@v4 to include persist-credentials: false and swap the three
action references (actions/checkout, oven-sh/setup-bun, actions/setup-node) to
their corresponding commit SHAs so the workflow is pinned and tokens aren't
persisted. Ensure you make the same changes for other jobs in ci.yml that use
actions/checkout@v4 without persist-credentials and any other mutable action
tags.


- name: Install dependencies
run: bun install

# The worker's exec-sandbox uses bwrap on Linux; ubuntu-latest blocks
# unprivileged user namespaces via AppArmor, so flip the sysctl too.
- name: Install bubblewrap (worker exec-sandbox)
run: |
sudo apt-get update
sudo apt-get install -y bubblewrap coreutils
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
bwrap --version

# Embedded Postgres (PG18) ICU shim — see the sdk-e2e job note above;
# scripts/cli-smoke.sh runs the same scripts/sdk-e2e/fix-embedded-pg-icu.mjs.

- name: Build all packages (CLI + server bundle + sdk + pgvector-embedded)
run: make build-packages

- name: CLI command-coverage smoke (every command runs)
run: bash scripts/cli-smoke.sh

# Frontend tests run under jsdom via vitest. owletto is a submodule;
# forks without the deploy key get a stub package and skip these.
frontend:
Expand Down Expand Up @@ -252,29 +298,29 @@
# the workspace symlink → dist. connector-worker itself is required
# because `packages/server/src/utils/connector-catalog.ts` imports
# `@lobu/connector-worker/compile` (the shared compile pipeline),
# which resolves via the package's `exports` field to
# `./dist/compile/index.js`. Vitest's vite-node resolver follows
# the `import` condition and fails to load if dist is absent —
# which transitively breaks every integration file whose import
# graph touches `queue-helpers` / `worker-api` / `connector-catalog`.
#
# pgvector-embedded must build too: the test backend
# (src/__tests__/setup/embedded-postgres-backend.ts) imports
# `@lobu/pgvector-embedded`, so vite resolves its `import` condition
# (./dist/index.js) at transform time even though the embedded backend
# is never started when DATABASE_URL points at the external Postgres.
run: |
cd packages/core && bun run build && cd ../..
cd packages/connector-sdk && bun run build && cd ../..
cd packages/embeddings && bun run build && cd ../..
cd packages/connector-worker && bun run build && cd ../..
cd packages/pgvector-embedded && bun run build && cd ../..

- name: Verify Postgres health (fail fast if pgvector setup is broken)
run: |
for i in {1..20}; do
if pg_isready -h 127.0.0.1 -p 5433 -U postgres; then break; fi
sleep 1

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
done
PGPASSWORD=postgres psql -h 127.0.0.1 -p 5433 -U postgres -d lobu_test \
-c "CREATE EXTENSION IF NOT EXISTS vector"
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,4 @@ packages/*/~/
.lobu-dev/
.sdk-e2e-run/
.managed-e2e-run/
.cli-smoke-run/
10 changes: 9 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Development Makefile for Lobu

.PHONY: help setup build test clean dev build-packages ensure-submodule clean-workers test-unit test-integration test-e2e test-e2e-sdk typecheck task-setup task-clean e2e-browser bump review
.PHONY: help setup build test clean dev build-packages ensure-submodule clean-workers test-unit test-integration test-e2e test-e2e-sdk test-e2e-cli typecheck task-setup task-clean e2e-browser bump review

# Default target
help:
Expand All @@ -12,6 +12,7 @@ help:
@echo " make test-unit - Run the CI unit suite (no Postgres needed)"
@echo " make test-integration - Run the CI integration suite (needs DATABASE_URL with pgvector)"
@echo " make test-e2e - Boot the dev server + run openclaw-plugin e2e against it"
@echo " make test-e2e-cli - Boot lobu run + walk every CLI command (the CI cli-smoke gate)"
@echo " make clean-workers - Stop any running embedded worker subprocesses"
@echo " make typecheck - Strict typecheck (same as Dockerfile) for server + owletto"
@echo " make task-setup NAME=<name> - Create a paired worktree at .claude/worktrees/<name> (lobu + submodule on real branch, .env copied, ports auto-assigned, Lobu context registered)"
Expand Down Expand Up @@ -148,6 +149,13 @@ test-e2e:
test-e2e-sdk:
@./scripts/sdk-e2e.sh

# CLI command-coverage smoke: boots one `lobu run` (embedded Postgres + mock
# provider) under an isolated HOME and walks EVERY `lobu` command/subcommand
# once, asserting each runs (or fails gracefully). Self-contained, no key. This
# is the CI `cli-smoke` gate; run it locally the same way.
test-e2e-cli:
@./scripts/cli-smoke.sh

# Stop any embedded worker subprocesses left over from a crashed gateway.
# Workers are normally cleaned up when the gateway exits; this target is a
# safety net for orphaned bun processes spawned by EmbeddedDeploymentManager.
Expand Down
3 changes: 3 additions & 0 deletions config/biome.config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
"!**/workspaces/**",
"!**/.claude/debug/**",
"!**/.claude/worktrees/**",
"!**/.cli-smoke-run/**",
"!**/.sdk-e2e-run/**",
"!**/.managed-e2e-run/**",
"!**/dist/**",
"!**/src/errors.js",
"!**/*.raw.js",
Expand Down
Loading
Loading