feat(setup): clone v2 host-service DBs alongside v1 local DB#3630
Conversation
Seeds ~/.superset/host/<orgId>/host.db into superset-dev-data/host/<orgId>/ so v2 workspaces open with host-service state prefilled, matching the existing v1 local.db clone behavior.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a setup step to seed per-organization host SQLite DBs from Changes
Sequence Diagram(s)sequenceDiagram
participant Setup as Setup Script
participant Src as Source FS (~/.superset/host)
participant Dest as Dest FS (superset-dev-data/host)
participant DB as sqlite3
Setup->>Src: enumerate org dirs, locate host.db
alt no sources
Setup->>Setup: step_skipped (no host DBs)
else sources found
loop per org
Setup->>Dest: check existing host.db
alt FORCE_OVERWRITE_DATA=1
Setup->>Dest: remove existing host.db, -shm, -wal
else
Setup->>Setup: skip org (exists)
end
Src->>Dest: copy host.db (+ -shm/-wal if present)
Dest->>DB: PRAGMA wal_checkpoint(TRUNCATE)
DB-->>Dest: checkpoint result
end
Setup->>Setup: report counts (copied/skipped) or fail on error
end
sequenceDiagram
participant Browser as Browser (DashboardLayout)
participant Hook as useDevSeedV2Sidebar
participant API as Workspace Store / Collections
participant Sidebar as Sidebar Manager (ensureWorkspaceInSidebar)
Browser->>Hook: invoke on layout mount
Hook->>Hook: check NODE_ENV and localStorage flag
alt not dev or already seeded
Hook-->>Browser: no-op
else
Hook->>API: read accessibleWorkspaces and local collection state
alt local state has entries
Hook->>Hook: set localStorage flag, no pinning
else
loop per workspace
Hook->>Sidebar: ensureWorkspaceInSidebar(workspace.id, workspace.projectId)
end
Hook->>Hook: set localStorage flag "1"
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR extends the dev-environment setup script to clone v2 host-service SQLite databases (one per org) from Key changes:
Confidence Score: 5/5Safe to merge — the new step is a well-scoped dev-only utility that mirrors an established pattern and degrades gracefully when sources are absent. The implementation is a faithful port of the existing No files require special attention.
|
| Filename | Overview |
|---|---|
| .superset/lib/setup/main.sh | Correctly inserts step_seed_host_dbs as Step 5 between the local-DB seed and the auth-token seed; minor duplicate "Step 6" comment on the Neon branch step. |
| .superset/lib/setup/steps.sh | Adds step_seed_host_dbs which faithfully mirrors the step_seed_local_db pattern: skips absent sources, respects -f/--force, clears stale WAL siblings before overwrite, copies main+shm+wal, and checkpoints the copy's WAL. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[setup_main] --> B[step_seed_local_db]
B -->|force=1| B1["rm -rf superset-dev-data/"]
B1 --> B2[Copy ~/.superset/local.db → superset-dev-data/local.db]
B --> C[step_seed_host_dbs]
C --> C1{~/.superset/host/ exists?}
C1 -->|No| C2[warn + step_skipped]
C1 -->|Yes| C3[Collect org dirs with host.db]
C3 --> C4{Any host.db found?}
C4 -->|No| C5[warn + step_skipped]
C4 -->|Yes| C6[mkdir -p superset-dev-data/host/]
C6 --> C7[For each org_id]
C7 --> C8{dest exists & not force?}
C8 -->|Yes| C9[warn + skip org]
C8 -->|No| C10[rm -f stale WAL siblings if force]
C10 --> C11[cp host.db + -shm + -wal]
C11 --> C12[sqlite3 PRAGMA wal_checkpoint TRUNCATE]
C12 --> C7
C7 -->|done| C13[success seeded/skipped count]
C13 --> D[step_seed_auth_token]
Comments Outside Diff (1)
-
.superset/lib/setup/main.sh, line 49 (link)Duplicate step number in comment
After inserting the new Step 5 (Seed host-service DBs), the Neon branch step is still labeled "Step 6", which collides with the auth-token step above it.
Prompt To Fix With AI
This is a comment left during a code review. Path: .superset/lib/setup/main.sh Line: 49 Comment: **Duplicate step number in comment** After inserting the new Step 5 (Seed host-service DBs), the Neon branch step is still labeled "Step 6", which collides with the auth-token step above it. How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: .superset/lib/setup/main.sh
Line: 49
Comment:
**Duplicate step number in comment**
After inserting the new Step 5 (Seed host-service DBs), the Neon branch step is still labeled "Step 6", which collides with the auth-token step above it.
```suggestion
# Step 7: Setup Neon branch
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "feat(setup): clone v2 host-service DBs a..." | Re-trigger Greptile
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.superset/lib/setup/main.sh (1)
44-74:⚠️ Potential issue | 🟡 MinorRenumber remaining step comments — duplicate "Step 6" and off-by-one after the insertion.
You bumped auth-token to Step 6 (line 44), but the comments on lines 49, 54, 59, 64, and 69 still say Step 6/7/8/9/10. They should shift to 7/8/9/10/11 so the numbering stays unique and monotonic. Doc-only, no runtime impact.
📝 Proposed fix
- # Step 6: Setup Neon branch + # Step 7: Setup Neon branch if ! step_setup_neon_branch; then step_failed "Setup Neon branch" fi - # Step 7: Allocate port base (file-backed) + # Step 8: Allocate port base (file-backed) if ! allocate_port_base; then step_failed "Allocate port base" fi - # Step 8: Start Electric SQL + # Step 9: Start Electric SQL if ! step_start_electric; then step_failed "Start Electric SQL" fi - # Step 9: Write .env file + # Step 10: Write .env file if ! step_write_env; then step_failed "Write .env file" fi - # Step 10: Setup local MCP in .mcp.json (opt-in) + # Step 11: Setup local MCP in .mcp.json (opt-in) if [ "$SETUP_LOCAL_MCP" = "1" ]; then🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.superset/lib/setup/main.sh around lines 44 - 74, Update the inline step comments so they remain sequential after inserting step_seed_auth_token: change the comment above step_setup_neon_branch to "Step 7", allocate_port_base to "Step 8", step_start_electric to "Step 9", step_write_env to "Step 10", and the conditional for step_setup_local_mcp to "Step 11"; locate these near the calls to step_setup_neon_branch, allocate_port_base, step_start_electric, step_write_env, and step_setup_local_mcp in the same block that calls step_seed_auth_token and step_failed and adjust only the comment text.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In @.superset/lib/setup/main.sh:
- Around line 44-74: Update the inline step comments so they remain sequential
after inserting step_seed_auth_token: change the comment above
step_setup_neon_branch to "Step 7", allocate_port_base to "Step 8",
step_start_electric to "Step 9", step_write_env to "Step 10", and the
conditional for step_setup_local_mcp to "Step 11"; locate these near the calls
to step_setup_neon_branch, allocate_port_base, step_start_electric,
step_write_env, and step_setup_local_mcp in the same block that calls
step_seed_auth_token and step_failed and adjust only the comment text.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b968aed0-b9b9-4fa4-b545-587a5b6a709b
📒 Files selected for processing (2)
.superset/lib/setup/main.sh.superset/lib/setup/steps.sh
🧹 Preview Cleanup CompleteThe following preview resources have been cleaned up:
Thank you for your contribution! 🎉 |
The v2 sidebar reads pinned-workspace state from renderer localStorage (`v2WorkspaceLocalState`, etc.), which lives in the per-app userData dir. Each dev worktree gets a fresh `Superset (<workspace>)` userData → empty sidebar even when the cloud + host.db clones have plenty of accessible workspaces. Copy `Local Storage/` from the prod app's userData using APFS clonefile so prod can keep running.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.superset/lib/setup/steps.sh:
- Around line 663-681: When a copy in the for-loop (in
.superset/lib/setup/steps.sh around the block using source_db, dest_db and
copy_failed) fails you must remove any partially copied dest files so a future
run doesn't see a lone host.db; modify the error path so when copy_failed is set
(or immediately after a cp fails inside the loop) you delete dest_file and also
remove any of "${dest_db}", "${dest_db}-shm", and "${dest_db}-wal" that may have
been created before returning 1, then return the non-zero status as before.
- Around line 730-732: The removal block that runs when force_overwrite is "1"
currently ignores failures from rm -rf and must fail fast: after attempting to
remove "$dest_local_storage" (when [ -d "$dest_local_storage" ] && [
"$force_overwrite" = "1" ]), check the rm exit status and if it failed, emit an
error and exit with a non‑zero status so the script does not continue
(preventing mixing production Local Storage with stale files); update the block
around dest_local_storage and force_overwrite to perform this check and exit on
failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 175c2d5a-7163-4a5b-ba71-18dce0352ac9
📒 Files selected for processing (2)
.superset/lib/setup/main.sh.superset/lib/setup/steps.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- .superset/lib/setup/main.sh
| if [ -d "$dest_local_storage" ] && [ "$force_overwrite" = "1" ]; then | ||
| rm -rf "$dest_local_storage" | ||
| fi |
There was a problem hiding this comment.
Fail if forced renderer-state deletion fails.
With -f, continuing after a failed rm -rf can merge prod Local Storage into stale LevelDB files instead of replacing them.
🛠️ Proposed fix
if [ -d "$dest_local_storage" ] && [ "$force_overwrite" = "1" ]; then
- rm -rf "$dest_local_storage"
+ if ! rm -rf "$dest_local_storage"; then
+ error "Failed to remove existing renderer state at $dest_local_storage"
+ return 1
+ fi
fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if [ -d "$dest_local_storage" ] && [ "$force_overwrite" = "1" ]; then | |
| rm -rf "$dest_local_storage" | |
| fi | |
| if [ -d "$dest_local_storage" ] && [ "$force_overwrite" = "1" ]; then | |
| if ! rm -rf "$dest_local_storage"; then | |
| error "Failed to remove existing renderer state at $dest_local_storage" | |
| return 1 | |
| fi | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.superset/lib/setup/steps.sh around lines 730 - 732, The removal block that
runs when force_overwrite is "1" currently ignores failures from rm -rf and must
fail fast: after attempting to remove "$dest_local_storage" (when [ -d
"$dest_local_storage" ] && [ "$force_overwrite" = "1" ]), check the rm exit
status and if it failed, emit an error and exit with a non‑zero status so the
script does not continue (preventing mixing production Local Storage with stale
files); update the block around dest_local_storage and force_overwrite to
perform this check and exit on failure.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".superset/lib/setup/steps.sh">
<violation number="1" location=".superset/lib/setup/steps.sh:731">
P2: If `rm -rf "$dest_local_storage"` fails (e.g. permissions, locked file), the script silently continues to `cp`, merging new LevelDB files on top of stale ones instead of cleanly replacing them. Check the return code and bail out on failure.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Drop the renderer-state file-copy approach (Chromium localStorage is per-origin: dev runs on http://localhost:<port>, prod is file://, so copying the leveldb seeds the wrong namespace). Instead, add a small dev-only hook in the dashboard layout that pins every accessible v2 workspace once per worktree's userData. A localStorage flag prevents re-pinning workspaces the user has explicitly unpinned later.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts (1)
21-29: MoveuseAccessibleV2Workspaces()behind a dev-only guard to prevent database queries in production.
useAccessibleV2Workspaces()callsuseLiveQuery(), which subscribes to database changes and executes on every render. Even though the seeding effect returns early in production, this hook runs before theNODE_ENVcheck, adding unnecessary database overhead outside development. Either wrap this component in a dev-only condition, or pass a conditional flag to prevent the query execution in production.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts` around lines 21 - 29, The call to useAccessibleV2Workspaces() runs its useLiveQuery subscription even in production; move or gate it so it only runs in development: inside useDevSeedV2Sidebar, defer calling useAccessibleV2Workspaces() (and thus useLiveQuery) until after the env.NODE_ENV === "development" check (or modify useAccessibleV2Workspaces to accept a boolean skip flag and pass skip = env.NODE_ENV !== "development"); update references to accessibleWorkspaces/ensureWorkspaceInSidebar/collections accordingly so the effect early-returns when SEED_FLAG_KEY is set but no DB query runs in production.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In
`@apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts`:
- Around line 21-29: The call to useAccessibleV2Workspaces() runs its
useLiveQuery subscription even in production; move or gate it so it only runs in
development: inside useDevSeedV2Sidebar, defer calling
useAccessibleV2Workspaces() (and thus useLiveQuery) until after the env.NODE_ENV
=== "development" check (or modify useAccessibleV2Workspaces to accept a boolean
skip flag and pass skip = env.NODE_ENV !== "development"); update references to
accessibleWorkspaces/ensureWorkspaceInSidebar/collections accordingly so the
effect early-returns when SEED_FLAG_KEY is set but no DB query runs in
production.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 96bae676-6d59-49a5-86db-1a0f00f08a5e
📒 Files selected for processing (5)
.superset/lib/setup/main.sh.superset/lib/setup/steps.shapps/desktop/src/renderer/routes/_authenticated/_dashboard/layout.tsxapps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/index.tsapps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts
✅ Files skipped from review due to trivial changes (1)
- apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/index.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- .superset/lib/setup/main.sh
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".superset/lib/setup/main.sh">
<violation number="1" location=".superset/lib/setup/main.sh:44">
P1: `step_seed_renderer_state` is missing from `setup_main`, so renderer Local Storage seeding no longer runs before auth token seeding.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| step_failed "Seed host-service DBs" | ||
| fi | ||
|
|
||
| # Step 6: Seed auth token into superset-dev-data/ |
There was a problem hiding this comment.
P1: step_seed_renderer_state is missing from setup_main, so renderer Local Storage seeding no longer runs before auth token seeding.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .superset/lib/setup/main.sh, line 44:
<comment>`step_seed_renderer_state` is missing from `setup_main`, so renderer Local Storage seeding no longer runs before auth token seeding.</comment>
<file context>
@@ -41,12 +41,7 @@ setup_main() {
- fi
-
- # Step 7: Seed auth token into superset-dev-data/
+ # Step 6: Seed auth token into superset-dev-data/
if ! step_seed_auth_token; then
step_failed "Seed auth token"
</file context>
| # Step 6: Seed auth token into superset-dev-data/ | |
| # Step 6: Seed Electron renderer Local Storage from prod userData | |
| if ! step_seed_renderer_state; then | |
| step_failed "Seed renderer state" | |
| fi | |
| # Step 7: Seed auth token into superset-dev-data/ |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts (1)
17-36: LGTM — dev-only seeding logic is well-guarded.The ordering of guards (NODE_ENV → flag → empty accessible list → existing local state) is correct, and setting
SEED_FLAG_KEYwhen pre-existing v2 sidebar state is detected cleanly avoids clobbering a user who has already curated their sidebar. The docstring nicely explains the per-origin localStorage rationale.One optional nit:
collectionsis the entire context object and will almost certainly have a new identity on each provider render, so this effect re-runs frequently. It's harmless because the early-return on the flag short-circuits immediately, but you could narrow the dependency tocollections.v2WorkspaceLocalState.state.size(or read it via a ref) to make the intent clearer and avoid needlessly re-executing the guards.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts` around lines 17 - 36, The effect in useDevSeedV2Sidebar re-runs unnecessarily because the dependency includes the whole collections object; change the dependency to the specific value you care about (collections.v2WorkspaceLocalState.state.size) or read that size into a ref and depend on the ref instead so the effect only reruns when the local-state size changes; keep accessibleWorkspaces and ensureWorkspaceInSidebar in the dependency array and leave SEED_FLAG_KEY and the logic in useDevSeedV2Sidebar unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In
`@apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts`:
- Around line 17-36: The effect in useDevSeedV2Sidebar re-runs unnecessarily
because the dependency includes the whole collections object; change the
dependency to the specific value you care about
(collections.v2WorkspaceLocalState.state.size) or read that size into a ref and
depend on the ref instead so the effect only reruns when the local-state size
changes; keep accessibleWorkspaces and ensureWorkspaceInSidebar in the
dependency array and leave SEED_FLAG_KEY and the logic in useDevSeedV2Sidebar
unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c0f22458-1ba5-4396-985e-445dba42f5d5
📒 Files selected for processing (1)
apps/desktop/src/renderer/routes/_authenticated/hooks/useDevSeedV2Sidebar/useDevSeedV2Sidebar.ts
If host.db copies but a -wal/-shm sibling fails, leaving the lone host.db in place would make the next non-force run skip the org as "already seeded." Wipe the per-org dest before returning the error. Caught by coderabbitai on PR #3630.
1 participant
Summary
step_seed_host_dbsto.superset/setup.shthat clones every~/.superset/host/<orgId>/host.dbintosuperset-dev-data/host/<orgId>/host.dbso v2 workspaces open with host-service state prefilled (projects, workspaces, terminals, PRs).local.dbclone: copies-shm/-walsiblings, checkpoints the WAL on the copy, respects-f/--force, skips when source is absent.step_seed_local_dband beforestep_seed_auth_tokeninmain.sh(so the force-overwriterm -rf superset-dev-data/from the v1 seed happens first).Test plan
step_seed_host_dbsin this worktree — 3 orgs copied, resulting DBs open with expected host-service tables (projects,workspaces,terminal_sessions,pull_requests,__drizzle_migrations).host.dbare skipped; rerun without-fskips existing destinations; rerun with-foverwrites and clears stale WAL siblings.Summary by cubic
Clone v2 host-service DBs and auto-pin accessible v2 workspaces in dev so v2 opens with prefilled data and a populated sidebar.
New Features
step_seed_host_dbsto clone per-orghost.dbfrom~/.superset/host/<orgId>/tosuperset-dev-data/host/<orgId>/; copies-shm/-wal, WAL checkpoints, and runs after the v1 local DB seed and before auth token seeding.useDevSeedV2Sidebarto pin all accessible v2 workspaces once per worktree using a localStorage flag, avoiding per-origin localStorage issues.Bug Fixes
host.dbisn’t left behind, preventing false “already seeded” skips on the next run.Written for commit 91d5f10. Summary will update on new commits.
Summary by CodeRabbit
New Features
Improvements
Bug Fixes