feat: OSM-powered native trail data layer (POC)#2306
Merged
Conversation
Adds a native trail data layer powered by OpenStreetMap data. Raw OSM data is imported via osm2pgsql and queried intelligently at runtime using PostGIS — no custom transformation or sync burden. Infrastructure: - infrastructure/osm/hiking.lua — osm2pgsql flex output Lua config; imports hiking ways (path/footway/track) and named route relations with member JSONB for runtime stitching fallback - infrastructure/osm/import.sh — one-command import; defaults to Utah extract (~150 MB) from Geofabrik for POC testing Database: - Migration 0037: enables PostGIS, creates hiking_ways and hiking_relations tables with GIST spatial indexes - trips.trail_osm_id — lightweight OSM relation ref (no FK, OSM data is external) API (all under /api/trails, default off behind enableTrails flag): - GET /trails/search?q=&lat=&lon=&radius= — text + ST_DWithin spatial search over hiking_relations, returns lightweight result list - GET /trails/:osmId — trail metadata without geometry - GET /trails/:osmId/geometry — full GeoJSON; uses pre-built geometry from osm2pgsql when available, falls back to runtime ST_LineMerge stitching from member ways and caches the result back - POST /trails/alltrails-preview — server-side OG tag extraction from an AllTrails URL; no API key needed, SSR pages include OG in HTML Config: - enableTrails feature flag (default false) in packages/config https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Adds full test coverage for the trail feature (search, metadata, geometry
stitching, and AllTrails OG preview) so the suite runs cleanly in GitHub
Actions via the existing api-tests workflow.
Infrastructure:
- Dockerfile.test — extends pgvector/pgvector:pg15 with PostGIS 3 via apt,
giving the test DB both vector similarity and spatial query support
- docker-compose.test.yml — builds from Dockerfile.test instead of pulling
the pgvector image directly
Test setup:
- setup.ts — adds hiking_ways and hiking_relations to the per-test TRUNCATE
so OSM data is in a clean state for each test
- test/fixtures/trail-fixtures.ts — test geometry constants and fixture
builder functions; WKT segments in the Sierra Nevada for realism
- test/utils/osm-db-helpers.ts — seedHikingWay / seedHikingRelation helpers
that insert PostGIS geometry rows via raw SQL (tables are not in Drizzle
schema, they are osm2pgsql-managed in production)
Tests (test/trails.test.ts — 18 cases):
- GET /trails/search: text search, case-insensitive, empty results,
spatial search via ST_DWithin, combined text+spatial, bbox presence
- GET /trails/:osmId: metadata fields, bbox, 404 and 400 error paths
- GET /trails/:osmId/geometry: pre-built geometry, runtime ST_LineMerge
stitching from member ways, multi-way merge, cache write-back, error paths
- POST /trails/alltrails-preview: SSRF guard, AllTrails-only allow-list,
OG tag extraction (both attribute orders), 422 on missing title, 502 on
upstream error — all using vi.stubGlobal('fetch', ...) for fetch mocking
https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Warning
|
| Cohort / File(s) | Summary |
|---|---|
Query Key Centralization apps/admin/app/dashboard/page.tsx, apps/admin/app/dashboard/users/page.tsx, apps/admin/app/dashboard/packs/page.tsx, apps/admin/app/dashboard/catalog/page.tsx, apps/admin/components/edit-catalog-dialog.tsx, apps/admin/hooks/use-catalog-analytics.ts, apps/admin/hooks/use-platform-analytics.ts, apps/admin/lib/queryKeys.ts |
Introduces centralized queryKeys registry for React Query, replacing hardcoded array-based keys with strongly-typed factory functions across admin dashboard and hooks for consistent cache management. |
CF Access Authentication Migration apps/admin/app/login/page.tsx, apps/admin/components/auth-guard.tsx, apps/admin/lib/cfAccess.ts, packages/api/src/middleware/cfAccess.ts |
Transitions from one-time async CF Access checks to hook-driven identity lookups using TanStack Query; simplifies auth-guard logic and adds Zod schema validation for JWT payloads. |
OSM Hiking Infrastructure infrastructure/osm/hiking.lua, infrastructure/osm/import.sh |
Adds osm2pgsql flex script for importing hiking trails from OpenStreetMap PBF files into PostGIS tables, with standalone ingestion script supporting Geofabrik download and database insertion. |
Database Schema & Migrations packages/api/drizzle/0037_osm_hiking_tables.sql, packages/api/drizzle/meta/0037_snapshot.json, packages/api/drizzle/meta/_journal.json, packages/api/src/db/schema.ts |
Extends database schema with hiking_ways and hiking_relations PostGIS tables for trail storage, adds trail_osm_id to trips table, and includes Drizzle migration snapshots. |
Trails API Implementation packages/api/src/routes/trails/index.ts, packages/api/src/routes/index.ts |
Introduces new trails API endpoints: /trails/search with spatial/text filtering, /trails/:osmId for metadata, /trails/:osmId/geometry with dynamic geometry stitching, and /trails/alltrails-preview for OG metadata extraction. |
Test Infrastructure & Coverage packages/api/test/fixtures/trail-fixtures.ts, packages/api/test/utils/osm-db-helpers.ts, packages/api/test/trails.test.ts, packages/api/test/setup.ts, packages/api/Dockerfile.test, packages/api/docker-compose.test.yml |
Adds test utilities for seeding hiking data, comprehensive trails API tests covering spatial queries and geometry stitching, and Docker test infrastructure with PostGIS support. |
Admin Routes Cleanup packages/api/src/routes/admin/index.ts |
Removes HTML/HTMX server-rendered UI endpoints, consolidates authentication middleware to return JSON errors, and adds query parameter support for list endpoint filtering. |
Feature Configuration packages/config/src/config.ts |
Adds EnableTrails feature flag to control trails functionality deployment. |
Sequence Diagrams
sequenceDiagram
participant Client as Client/Browser
participant AuthGuard as AuthGuard Component
participant Hook as useCFAccessIdentity Hook
participant TQ as TanStack Query
participant API as CF Access API
participant Server as Login/Protected Routes
Client->>AuthGuard: Mount protected page
AuthGuard->>Hook: Call useCFAccessIdentity()
Hook->>TQ: useQuery with cached result
TQ->>API: GET /cdn-cgi/access/get-identity
API-->>TQ: Return JWT payload
TQ->>Hook: Validate with Zod schema
Hook-->>AuthGuard: Return {data, isPending}
alt isPending is true
AuthGuard->>Client: Render nothing (loading)
else cfIdentity exists
AuthGuard->>Server: Render protected content
else No CF identity & no auth token
AuthGuard->>Server: Redirect to /login
end
sequenceDiagram
participant Client as Client
participant API as Trails API
participant DB as PostgreSQL + PostGIS
participant AllTrails as AllTrails.com
Client->>API: GET /trails/search?lat=X&lon=Y&radius=R&q=name
API->>DB: SELECT with ST_DWithin + ILIKE filters
DB-->>API: Return lightweight trail results with bbox
API-->>Client: Return {id, name, bbox, distance}
Client->>API: GET /trails/:osmId/geometry
API->>DB: Check if geometry cached
alt Geometry exists
DB-->>API: Return stored GeoJSON
else Geometry is null
API->>DB: SELECT member ways and collect geometries
DB-->>API: Return member LineStrings
API->>API: ST_LineMerge to stitch ways
API->>DB: UPDATE hiking_relations with merged geometry
DB-->>API: Confirm cached_at timestamp
end
API-->>Client: Return GeoJSON geometry
Client->>API: POST /trails/alltrails-preview with URL
API->>AllTrails: Server-side fetch HTML
AllTrails-->>API: Return HTML page
API->>API: Extract OG metadata via regex
API-->>Client: Return {title, description, image}
Estimated code review effort
🎯 4 (Complex) | ⏱️ ~50 minutes
Suggested labels
web, api, database
Suggested reviewers
- Isthisanmol
Poem
🐰 Hops through the OSM forest
Query keys bundled tight,
Auth flows simplified with hooks,
Trails dance in PostGIS light,
Geography and hiking dreams take flight! 🥾✨
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (4 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The title accurately summarizes the main refactoring objective: migrating CF Access identity fetch logic to TanStack Query. |
| Linked Issues check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
claude/osm-trail-poc-TJrsB
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
Contributor
Coverage Report for API Unit Tests Coverage (./packages/api)
File Coverage
|
||||||||||||||||||||||||||||||||||||||
Contributor
Coverage Report for Expo Unit Tests Coverage (./apps/expo)
File CoverageNo changed files found. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request introduces initial backend + test infrastructure for OSM-backed “Trails” functionality in the API (PostGIS tables/migration, new /api/trails/* routes, and integration tests), plus supporting infra for importing OSM hiking data via osm2pgsql.
Changes:
- Add PostGIS-backed OSM hiking tables (
hiking_ways,hiking_relations) via Drizzle migration and wire new/api/trailsroutes (search, metadata, geometry stitching+cache, AllTrails OG preview). - Add Vitest integration coverage for the trails routes, including raw-SQL seed helpers for OSM tables and test DB cleanup updates.
- Update test Postgres container to include PostGIS (custom Dockerfile), and add osm2pgsql flex config + import script under
infrastructure/osm.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/config/src/config.ts | Adds EnableTrails feature flag default configuration. |
| packages/api/src/routes/trails/index.ts | New Trails route group: search, metadata, geometry stitching/caching, AllTrails OG preview. |
| packages/api/src/routes/index.ts | Registers trailsRoutes under /api. |
| packages/api/src/db/schema.ts | Adds trips.trailOsmId column to schema. |
| packages/api/drizzle/0037_osm_hiking_tables.sql | Migration enabling PostGIS + creating OSM hiking tables + adding trips.trail_osm_id. |
| packages/api/drizzle/meta/_journal.json | Registers migration 0037 in Drizzle journal. |
| packages/api/drizzle/meta/0037_snapshot.json | New Drizzle snapshot reflecting migration 0037 state. |
| packages/api/test/trails.test.ts | New integration tests for Trails endpoints and AllTrails preview. |
| packages/api/test/utils/osm-db-helpers.ts | Raw-SQL seed helpers for hiking_ways/hiking_relations with PostGIS geometry literals. |
| packages/api/test/fixtures/trail-fixtures.ts | Shared fixture types/constants + factory helpers for trail test data. |
| packages/api/test/setup.ts | Ensures OSM hiking tables are truncated between tests. |
| packages/api/docker-compose.test.yml | Switches test DB to a locally built Postgres image (to include PostGIS). |
| packages/api/Dockerfile.test | Extends pgvector/pg15 image to install PostGIS packages. |
| infrastructure/osm/hiking.lua | osm2pgsql flex config for importing hiking ways + route relations into PostGIS tables. |
| infrastructure/osm/import.sh | Script to run osm2pgsql import into Neon, with optional PBF download. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| return status(400, { error: 'Invalid URL' }); | ||
| } | ||
|
|
||
| if (!parsed.hostname.endsWith('alltrails.com')) { |
Comment on lines
+328
to
+337
| try { | ||
| const response = await fetch(url, { | ||
| headers: { | ||
| 'User-Agent': | ||
| 'Mozilla/5.0 (compatible; PackRat/1.0; +https://packrat.world)', | ||
| Accept: 'text/html', | ||
| }, | ||
| // Cloudflare Workers fetch has no keepalive issues | ||
| signal: AbortSignal.timeout(8000), | ||
| }); |
Comment on lines
+1
to
+4
| import { createDb } from '@packrat/api/db'; | ||
| import { sql } from 'drizzle-orm'; | ||
| import { Elysia, status } from 'elysia'; | ||
| import { z } from 'zod'; |
Comment on lines
+56
to
+63
| osmId: overrides.osmId, | ||
| name: overrides.name ?? 'Test Hiking Way', | ||
| surface: overrides.surface ?? 'dirt', | ||
| difficulty: overrides.difficulty ?? null, | ||
| access: overrides.access ?? null, | ||
| foot: overrides.foot ?? 'yes', | ||
| geometryWkt: overrides.geometryWkt ?? DEFAULT_WAY_WKT, | ||
| }; |
Comment on lines
+69
to
+76
| name: overrides.name ?? 'Test Hiking Trail', | ||
| network: overrides.network ?? 'lwn', | ||
| distance: overrides.distance ?? '5 km', | ||
| difficulty: overrides.difficulty ?? 'easy', | ||
| description: overrides.description ?? null, | ||
| members: overrides.members ?? [], | ||
| geometryWkt: | ||
| overrides.geometryWkt !== undefined ? overrides.geometryWkt : DEFAULT_RELATION_WKT, |
| ) AS geojson | ||
| FROM hiking_ways | ||
| JOIN unnest( | ||
| ARRAY[${sql.raw(wayRefs.join(','))}]::bigint[] |
| await db.execute(sql` | ||
| UPDATE hiking_relations | ||
| SET | ||
| geometry = ST_GeomFromGeoJSON(${JSON.stringify(geometry)}), |
andrew-bierman
changed the title
- Remove unused osmId parameter from stitchTrailGeometry (violated complexity.useMaxParams: max 2) - Replace string concatenation with template literal in ILIKE query (style.useTemplate) - Apply Biome formatter fixes across trail routes, test fixtures, seeding helpers, and test file (line wrapping, db.execute call style) - Merge split imports from same source into single import in osm-db-helpers.ts https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
github-actions
Bot
added
the
dependencies
- Cast result.rows via unknown before TrailSearchResult[] to satisfy TS (Record<string,unknown>[] doesn't structurally overlap the interface) - Widen nullable fixture interface fields to string | null so ?? null fallbacks in makeHikingWay / makeHikingRelation type-check correctly https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
- Format migration JSON files to satisfy Biome formatter - Refactor verifyCFAccessRequest to use options object (fixes useMaxParams) - Replace interface casts with Zod schema parsing in trails route - Auto-fix env-validation.ts formatting https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Conflict in cfAccess.ts resolved by keeping the options-object refactor (our branch) over the biome-ignore suppress comment (development branch). https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
- Fix AllTrails hostname check: dot-boundary guard prevents notalltrails.com (was endsWith, now hostname === or endsWith dot-prefixed) - Validate response.url after redirects to prevent open-redirect SSRF - Wrap stitched geometry with ST_Multi() before caching to satisfy the geometry(MultiLineString) column typmod - Parameterize wayRefs with sql.join() instead of sql.raw() interpolation - Update cfAccess unit tests to use the new options-object signature https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Pre-existing TS2532 errors brought in from development branch merge. https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
infrastructure/osm/ was an orphan directory outside the workspace. Moved hiking.lua and import.sh into @packrat/osm-import so the tooling is a first-class workspace member with named scripts. https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Contributor
Deploying packrat-guides with
|
| Latest commit: |
bb4af65
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://69a7340b.packrat-guides-6gq.pages.dev |
| Branch Preview URL: | https://claude-osm-trail-poc-tjrsb.packrat-guides-6gq.pages.dev |
Contributor
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
packrat-admin | 4206538 | Commit Preview URL Branch Preview URL |
May 01 2026, 01:59 AM |
Contributor
Deploying packrat-landing with
|
| Latest commit: |
bb4af65
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://1505d9d6.packrat-landing.pages.dev |
| Branch Preview URL: | https://claude-osm-trail-poc-tjrsb.packrat-landing.pages.dev |
…ed_at - Rename hiking_ways → osm_ways, hiking_relations → osm_routes so the same tables serve hiking, cycling, skiing, and any future sport - Add sport column (set at import time by Lua config) instead of sport-specific columns like foot/access - Remove cached_at and geometry write-back — GET endpoints stay pure reads - Replace hiking.lua with routes.lua covering hiking, cycling, and skiing - Add sport= filter to the search endpoint - Rename fixtures/helpers to OsmWay/OsmRoute to match new schema https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
…d trail routes - Add @packrat/osm-db: standalone Drizzle package for the OSM database with PostGIS schema (osm_ways, osm_routes), GiST/GIN indexes, and pg_trgm support. Migrations split: 0000_extensions (hand-written CREATE EXTENSION) + 0001_osm_schema (drizzle-kit generated including geography casts and gin_trgm_ops operator classes). - Add packages/osm-import/import.ts: TypeScript importer replacing import.sh. Downloads PBF, runs osm2pgsql via Bun.spawn, verifies row counts against OSM_DATABASE_URL. - API: createOsmDb() reads OSM_DATABASE_URL (required in prod). Trail routes refactored to use dedicated OSM DB; stitchRouteGeometry moved to packages/api/src/services/trails.ts. trailOsmId column fixed to bigint. drizzle.config.ts updated with tablesFilter to prevent osm_ways/osm_routes from being managed by the main app migrations. - MCP: four trail tools registered (search_trails, get_trail, get_trail_geometry, preview_alltrails_url). - Test: osm-db-helpers and trail fixtures updated to use createOsmDb().
Adds /dashboard/trails — enter an OSM relation ID to fetch geometry from the trails API and render the stitched GeoJSON on a Leaflet map. Useful for visually verifying that osm2pgsql stitching is correct. - TrailMap component (leaflet, react-leaflet) with ssr:false dynamic import - trailsFetch() helper + TrailGeometry type + getTrailGeometry() in api.ts - osm query key group in queryKeys.ts - Trail Viewer nav item in config/nav.ts
Adds sync.ts which pg_dumps osm_ways + osm_routes from the local PostGIS instance and pg_restores them into a managed PostgreSQL DB (Supabase, Neon, RDS, etc.). The dump uses custom format (compressed) with --clean --if-exists so it is safe to re-run against an existing managed DB. Setting MANAGED_DB_URL causes `bun run import` to call sync automatically after a successful import. Can also be run standalone with `bun run sync` to re-push without re-importing.
- OSM_DATABASE_URL + OSM_PRODUCTION_DATABASE_URL added to nodeEnvSchema and root .env.example (OSM section with comments) - Rename MANAGED_DB_URL → OSM_PRODUCTION_DATABASE_URL for clarity - import.ts + sync.ts now read env via @packrat/env/node instead of process.env directly - Remove package-local .env.example (root .env.example is canonical)
…_DATABASE_URL OSM_DATABASE_URL_LOCAL — local Docker PostGIS used during import (scratch DB) OSM_DATABASE_URL — managed production DB, matches the Worker's Hyperdrive binding Follows the NEON_DATABASE_URL / NEON_DATABASE_URL_READONLY suffix pattern. The Worker API's OSM_DATABASE_URL references are unchanged.
Default 800 MB stalls on continent-scale imports during the way-processing pass due to insufficient node cache. Set OSM_CACHE_MB=6000 (or ~40% of available RAM) when importing North America or larger extracts.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an OSM-backed trails POC across import tooling, a dedicated OSM DB schema, new /api/trails/* endpoints, and an admin “Trail Viewer” map page for validating geometry.
Changes:
- Introduces
@packrat/osm-import+@packrat/osm-dbfor importing and migratingosm_ways/osm_routes(PostGIS + pg_trgm). - Adds authenticated Trails API routes (search, metadata, geometry stitching, AllTrails OG preview) plus integration tests and test DB PostGIS support.
- Adds an Admin Trail Viewer page with Leaflet-based geometry rendering and supporting query keys/API client methods.
Reviewed changes
Copilot reviewed 49 out of 51 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| tsconfig.json | Updates TS config exclusions for new packages; removes some compiler options. |
| packages/osm-import/package.json | Adds new private workspace package for OSM import/sync tooling. |
| packages/osm-import/import.ts | Implements osm2pgsql import + post-import migration step + optional sync. |
| packages/osm-import/sync.ts | Adds pg_dump/pg_restore-based sync from local OSM DB to managed DB. |
| packages/osm-import/routes.lua | osm2pgsql flex config producing osm_ways and osm_routes. |
| packages/osm-db/package.json | Adds new private workspace package for OSM DB schema/migrations. |
| packages/osm-db/tsconfig.json | TS config for the OSM DB package. |
| packages/osm-db/src/schema.ts | Drizzle schema for osm_ways/osm_routes plus spatial/trgm indexes. |
| packages/osm-db/src/index.ts | Re-exports OSM DB schema. |
| packages/osm-db/drizzle.config.ts | Drizzle-kit config for OSM DB migrations. |
| packages/osm-db/migrate.ts | Migration runner supporting standard Postgres vs Neon serverless. |
| packages/osm-db/drizzle/0000_extensions.sql | Enables PostGIS + pg_trgm extensions for the OSM DB. |
| packages/osm-db/drizzle/0001_osm_schema.sql | Creates OSM tables and indexes in the OSM DB. |
| packages/osm-db/drizzle/meta/_journal.json | Drizzle metadata for OSM DB migrations. |
| packages/osm-db/drizzle/meta/0001_snapshot.json | Drizzle snapshot metadata for OSM DB schema. |
| packages/mcp/src/tools/trails.ts | Registers MCP tools for trail search/details/geometry and AllTrails preview. |
| packages/env/src/node.ts | Adds Node env vars for OSM DB URLs and import cache sizing. |
| packages/config/src/config.ts | Adds enableTrails feature flag (default false). |
| packages/api/wrangler.jsonc | Adds Hyperdrive binding placeholder for dedicated OSM DB. |
| packages/api/tsconfig.json | Adjusts TS path mappings to be explicitly relative. |
| packages/api/src/utils/env-validation.ts | Adds OSM DB env validation + Hyperdrive binding typing. |
| packages/api/src/utils/tests/env-validation.test.ts | Updates env-validation unit test to include OSM DB URL. |
| packages/api/src/index.ts | Enriches worker env with Hyperdrive connection string for OSM DB. |
| packages/api/src/db/index.ts | Adds createOsmDb() connection helper for the dedicated OSM DB. |
| packages/api/src/db/schema.ts | Adds trips.trail_osm_id column to link trips to OSM relations. |
| packages/api/src/services/trails.ts | Adds PostGIS stitching helper for routes with missing geometry. |
| packages/api/src/routes/trails/index.ts | Adds /api/trails/* endpoints including search + geometry + AllTrails scraping. |
| packages/api/src/routes/index.ts | Mounts the new trails route group under /api. |
| packages/api/src/routes/admin/index.ts | Updates CF Access verification call signature usage. |
| packages/api/src/middleware/cfAccess.ts | Refactors CF Access verifier to accept an options object. |
| packages/api/src/middleware/tests/cfAccess.test.ts | Updates CF Access tests for the new options object signature. |
| packages/api/drizzle/0037_trips_trail_osm_id.sql | Migration adding trail_osm_id column + index to trips. |
| packages/api/drizzle/meta/_journal.json | Registers the new 0037 migration in drizzle metadata. |
| packages/api/drizzle/meta/0037_snapshot.json | Updates drizzle snapshot metadata for the new trips column. |
| packages/api/drizzle.config.ts | Switches drizzle-kit generation to config-based; excludes OSM tables. |
| packages/api/package.json | Updates db:generate to use drizzle config file. |
| packages/api/docker-compose.test.yml | Switches test Postgres image to a custom build with PostGIS. |
| packages/api/Dockerfile.test | Adds PostGIS to the pgvector test image. |
| packages/api/test/utils/osm-db-helpers.ts | Adds raw SQL helpers to seed osm_ways/osm_routes with geometry. |
| packages/api/test/fixtures/trail-fixtures.ts | Adds fixtures for OSM ways/routes and test geometry constants. |
| packages/api/test/trails.test.ts | Adds integration test suite for trails endpoints and AllTrails OG parsing. |
| packages/api/test/setup.ts | Adds OSM tables to truncation list and mocks createOsmDb() in tests. |
| apps/admin/package.json | Adds Leaflet + react-leaflet dependencies for Trail Viewer UI. |
| apps/admin/lib/queryKeys.ts | Adds TanStack query keys for OSM trail viewer queries. |
| apps/admin/lib/api.ts | Adds typed admin API call to fetch trail geometry. |
| apps/admin/config/nav.ts | Adds “Trail Viewer” entry to admin navigation. |
| apps/admin/components/trail-map.tsx | Adds Leaflet map component to render GeoJSON geometry. |
| apps/admin/app/dashboard/trails/page.tsx | Adds admin page to load and visualize an OSM relation geometry. |
| biome.json | Excludes coverage output from Biome processing. |
| bun.lock | Locks new workspace packages and new Leaflet/react-leaflet dependencies. |
| .env.example | Documents OSM DB env vars for local import and optional production sync. |
Comments suppressed due to low confidence (1)
packages/api/test/setup.ts:636
tablesToCleannow includesosm_ways/osm_routes, but the test global setup only applies migrations frompackages/api/drizzleand does not create these osm2pgsql tables/extensions. This will cause the TRUNCATE to fail on a fresh test DB. Either create these tables in the test DB bootstrap (e.g. runpackages/osm-db/drizzle/*.sqlin global setup) or avoid truncating/seeding them unless they exist.
const tablesToClean = [
'one_time_passwords',
'refresh_tokens',
'auth_providers',
'weight_history',
'pack_items',
'pack_template_items',
'packs',
'pack_templates',
'trail_condition_reports',
'catalog_item_etl_jobs',
'etl_jobs',
'catalog_items',
'invalid_item_logs',
'reported_content',
'comment_likes',
'post_likes',
'post_comments',
'posts',
'trips',
'users',
'osm_ways',
'osm_routes',
];
const tableList = tablesToClean.map((t) => `"${t}"`).join(', ');
await testDb.execute(sql.raw(`TRUNCATE TABLE ${tableList} RESTART IDENTITY CASCADE`));
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
41
to
44
| .use(uploadRoutes) | ||
| .use(trailConditionsRoutes) | ||
| .use(trailsRoutes) | ||
| .use(wildlifeRoutes) |
There was a problem hiding this comment.
trailsRoutes is registered unconditionally under /api. The PR description mentions these endpoints are behind an enableTrails flag (default false), but there’s no gating here (and no enableTrails usage in the API code). Either add a runtime guard (don’t mount routes / return 404 when disabled) or update the PR description to match behavior.
Comment on lines
+83
to
+91
| // OSM / trail database — dedicated Postgres instance with PostGIS. | ||
| // Create via: wrangler hyperdrive create osm-db --connection-string="postgresql://..." | ||
| // Then replace the id below with the output ID. | ||
| "hyperdrive": [ | ||
| { | ||
| "binding": "OSM_HYPERDRIVE", | ||
| "id": "TODO_replace_with_hyperdrive_config_id" | ||
| } | ||
| ], |
There was a problem hiding this comment.
hyperdrive.id is left as "TODO_replace_with_hyperdrive_config_id". Since wrangler.jsonc is used by wrangler dev in the Vitest Workers pool, this placeholder can break local dev/tests and deploys. Either remove the binding from the committed config (document it elsewhere), or provide a real Hyperdrive ID via environment-specific config.
| return status(422, { error: 'Could not extract trail metadata from page' }); | ||
| } | ||
|
|
||
| return { title, description, image, url }; |
There was a problem hiding this comment.
After following a redirect, the response payload still returns the original url from the request body rather than the validated redirect target URL. This can return a non-canonical URL even though you already resolved and fetched the redirect; consider returning the final URL (redirectUrl.toString() / response.url) instead.
Suggested change
| return { title, description, image, url }; | |
| return { title, description, image, url: response.url }; |
Comment on lines
+104
to
+112
| console.log('\nRe-applying migrations to restore indexes...'); | ||
| const journalClient = new pg.Client({ connectionString: DB_URL }); | ||
| await journalClient.connect(); | ||
| try { | ||
| await journalClient.query(`DELETE FROM drizzle.__drizzle_migrations`); | ||
| } catch { | ||
| // Journal table doesn't exist yet on a brand-new database — that's fine. | ||
| } finally { | ||
| await journalClient.end(); |
There was a problem hiding this comment.
The post-import step deletes all rows from drizzle.__drizzle_migrations to force re-running migrations. This is risky long-term: future migrations may not be written to be safely re-runnable, and clearing the journal can hide partial/failed states. Prefer a dedicated, idempotent “ensure indexes/extensions” migration or a targeted SQL step for just the indexes you need to restore after osm2pgsql --create.
Comment on lines
+9
to
+12
| tablesFilter: ['!osm_ways', '!osm_routes'], | ||
| dbCredentials: { | ||
| url: process.env.NEON_DATABASE_URL ?? '', | ||
| }, |
There was a problem hiding this comment.
dbCredentials.url falls back to an empty string when NEON_DATABASE_URL isn’t set, which tends to produce confusing drizzle-kit errors. Consider throwing a clear error when the env var is missing (similar to packages/osm-db/drizzle.config.ts).
Comment on lines
+15
to
+16
| // For Cloudflare Workers: set to env.OSM_HYPERDRIVE.connectionString (Hyperdrive binding). | ||
| OSM_DATABASE_URL: z.string().url(), |
There was a problem hiding this comment.
OSM_DATABASE_URL is required in apiEnvSchema, which will cause worker startup / validate:cloudflare-env to fail in environments where trails are meant to be feature-flagged off. Consider making it optional and only requiring it when the trails routes are enabled/used (or when OSM_HYPERDRIVE is present and used to derive it).
Suggested change
| // For Cloudflare Workers: set to env.OSM_HYPERDRIVE.connectionString (Hyperdrive binding). | |
| OSM_DATABASE_URL: z.string().url(), | |
| // Optional here because trails may be disabled, and Workers can use | |
| // env.OSM_HYPERDRIVE.connectionString (Hyperdrive binding) instead. | |
| OSM_DATABASE_URL: z.string().url().optional(), |
Comment on lines
+3
to
+5
| import 'leaflet/dist/leaflet.css'; | ||
| import L from 'leaflet'; | ||
| import { useEffect, useRef } from 'react'; |
There was a problem hiding this comment.
Importing global CSS (leaflet/dist/leaflet.css) from a component usually violates Next.js App Router rules (global CSS must be imported from app/layout.tsx or a top-level global stylesheet). This is likely to break the admin build; move the Leaflet CSS import into apps/admin/app/globals.css (or layout) instead.
Comment on lines
+267
to
+297
| /** | ||
| * POST /api/trails/alltrails-preview | ||
| * | ||
| * Fetches an AllTrails URL server-side and extracts OpenGraph metadata. | ||
| */ | ||
| .post( | ||
| '/alltrails-preview', | ||
| async ({ body }) => { | ||
| const { url } = body; | ||
|
|
||
| let parsed: URL; | ||
| try { | ||
| parsed = new URL(url); | ||
| } catch { | ||
| return status(400, { error: 'Invalid URL' }); | ||
| } | ||
|
|
||
| const { hostname, protocol } = parsed; | ||
| if ( | ||
| protocol !== 'https:' || | ||
| (hostname !== 'alltrails.com' && !hostname.endsWith('.alltrails.com')) | ||
| ) { | ||
| return status(400, { error: 'Only https://alltrails.com URLs are supported' }); | ||
| } | ||
|
|
||
| const AT_UA = 'Mozilla/5.0 (compatible; PackRat/1.0; +https://packrat.world)'; | ||
|
|
||
| try { | ||
| let response = await fetch(url, { | ||
| headers: { 'User-Agent': AT_UA, Accept: 'text/html' }, | ||
| redirect: 'manual', |
There was a problem hiding this comment.
This endpoint duplicates the existing /api/alltrails/preview scraper logic (see packages/api/src/routes/alltrails.ts). To avoid drift and double maintenance, consider reusing the existing route (or extracting a shared OG-scrape helper) instead of maintaining two separate implementations.
Comment on lines
+41
to
+45
| const IMPORT_MODE = process.env.IMPORT_MODE ?? 'create'; | ||
| // Node cache in MB. Default (800) is fine for small extracts; use 4000-8000 for | ||
| // continent-scale imports to avoid extreme disk I/O during the way-processing pass. | ||
| const CACHE_MB = process.env.OSM_CACHE_MB ?? '800'; | ||
| const UTAH_PBF_URL = 'https://download.geofabrik.de/north-america/us/utah-latest.osm.pbf'; |
There was a problem hiding this comment.
CACHE_MB reads from process.env.OSM_CACHE_MB even though the package already uses nodeEnv for env validation. Using nodeEnv.OSM_CACHE_MB ?? '800' would keep env access consistent and ensure the numeric-only validation is actually applied.
Comment on lines
+172
to
+178
| let geometry: unknown = null; | ||
|
|
||
| if (row.geojson) { | ||
| geometry = JSON.parse(row.geojson); | ||
| } else if (row.members && row.members.length > 0) { | ||
| geometry = await stitchRouteGeometry(db, row.members); | ||
| } |
There was a problem hiding this comment.
The route docs/PR description mention stitching results being cached back to the DB, but this handler never writes the stitched geometry back to osm_routes (no UPDATE) — every request will restitch when geometry is NULL. Either implement the write-back (and tests that assert it) or adjust the endpoint description/expectations.
- trails/index.ts: return response.url (canonical post-redirect URL) instead of the original request URL in alltrails-preview response - env-validation.ts: make OSM_DATABASE_URL optional in apiEnvSchema so worker startup doesn't fail in environments where trails are disabled - db/index.ts: throw clear error in createOsmDb() when OSM_DATABASE_URL is absent (fail at call-time, not silently) - drizzle.config.ts: throw explicit error when NEON_DATABASE_URL is missing instead of silently passing empty string to drizzle-kit https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Three layers of changes:
1. setup.ts — Elysia non-AOT query/params unwrap fix: in aot:false mode
Elysia wraps Decode output in { value: ... } but never unwraps it for
query/params (only body). Added onBeforeHandle hook to unwrap both.
2. setup.ts — OSM table DDL in beforeAll: createOsmDb() is mocked to
return testDb, so osm_ways/osm_routes must exist there. Added IF NOT
EXISTS CREATE TABLE with PostGIS geometry columns.
3. trails.test.ts — realistic hardened tests: sport filter, pagination,
non-way member types ignored, missing ways → null geometry, disconnected
ways → MultiLineString, radius > 500 → 400, all using apiWithAuth.
- Move createOsmDb() inside try/catch in all three OSM handlers so a missing OSM_DATABASE_URL surfaces as 503 (not an uncaught exception) - Make OSM_DATABASE_URL optional in apiEnvSchema so the Worker starts cleanly when trail features are not configured - Add explicit null guard in createOsmDb() with descriptive error message - Fix AllTrails preview to return response.url (canonical post-redirect URL) - Move Leaflet CSS import from trail-map.tsx component to globals.css (Next.js App Router requires global CSS in layout/globals only) - Remove wrangler.jsonc hyperdrive block with TODO placeholder ID that breaks wrangler dev; replace with an instructional comment - Use nodeEnv.OSM_CACHE_MB in import.ts instead of raw process.env
Conflicts resolved: - packages/api/src/index.ts: kept OSM enrichEnv/Hyperdrive wiring - packages/api/src/routes/admin/index.ts: took dev Bearer JWT auth guard - packages/api/src/routes/trails/index.ts: kept OSM DB implementation - packages/api/src/utils/env-validation.ts: kept OSM_HYPERDRIVE binding - tsconfig.json: took dev ignoreDeprecations for Expo SDK 55 compat - bun.lock: regenerated
- admin/index.ts: update verifyCFAccessRequest call to use options object
signature {teamDomain, aud} instead of the removed 3-arg positional form
- admin/lib/api.ts: fix trailsFetch to call getAuthHeader() (not the
non-existent buildAuthHeaders())
- admin/components/trail-map.tsx: add explicit L.Layer type annotation to
eachLayer callback to fix implicit any
- tsconfig.json: exclude packages/overpass from root type check (standalone
package with own tsconfig and deps not hoisted to root)
https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
The packrat-admin Workers Build was failing with "Module not found: Can't
resolve 'leaflet'" because the Cloudflare CI environment did not have leaflet
installed in node_modules at build time.
- next.config.mjs: add webpack externals so webpack maps `import L from
'leaflet'` to `window.L` (both server and client passes) — build succeeds
even when the npm package is absent
- layout.tsx: load leaflet JS (afterInteractive) and CSS from unpkg CDN so
the map works at runtime
- globals.css: remove `@import "leaflet/dist/leaflet.css"` which postcss-import
would also fail to resolve when the package is not installed
The TrailMap component uses `dynamic(..., { ssr: false })` so leaflet is
only accessed client-side; by the time the user submits an OSM ID the
afterInteractive script has loaded.
https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
Auto-generated by `next build`; provides TypeScript type references for Next.js. https://claude.ai/code/session_01LJnh37hSqTY8VMsqNyVRSb
- node.ts: add IMPORT_MODE to schema + parse call; wire OSM env vars that were declared in schema but missing from the parse object - import.ts: use nodeEnv.IMPORT_MODE instead of raw process.env access - osm-db/drizzle.config.ts, migrate.ts: import nodeEnv instead of process.env - api/drizzle.config.ts: import nodeEnv instead of process.env - trails/index.ts: replace dynamic new RegExp() with static top-level OG regexes - no-raw-regex: remove trails/index.ts exception (no longer needed) - no-raw-process-env: remove drizzle/migrate exceptions; keep import.ts for legitimate Bun.spawn OS env spreading
3 tasks
This was referenced May 1, 2026
andrew-bierman
added a commit
that referenced
this pull request
May 14, 2026
feat: OSM-powered native trail data layer (POC)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
osm2pgsqlflex output — no custom transformation, no sync burdenST_DWithinradius filter, andST_LineMergestitching of member ways into a continuous GeoJSON linestringWhat's in this PR
Infrastructure (
infrastructure/osm/)hiking.lua— osm2pgsql flex Lua config; importshighway=path/footway/trackways andtype=route, route=hikingrelations with member JSONB for stitching fallbackimport.sh— one-command import; auto-downloads Utah extract (~150 MB) from Geofabrik if no PBF is passedDatabase (migration
0037)CREATE EXTENSION IF NOT EXISTS postgis— enables PostGIS on Neonhiking_waystable — LineString geometry + GIST indexhiking_relationstable — MultiLineString geometry + GIST index +members jsonb+cached_atfor stitching cachetrips.trail_osm_id bigint— soft link to an OSM relation (intentionally no FK)API (all under
/api/trails, behindenableTrailsflag, defaultfalse)GET /trails/search?q=&lat=&lon=&radius=ST_DWithinspatial search; no geometry returnedGET /trails/:osmIdGET /trails/:osmId/geometryST_LineMerge(ST_Collect(...))frommembersJSONB and writes result back tocached_atPOST /trails/alltrails-preview{title, description, image, url}from OG tagsTests (18 cases across
test/trails.test.ts)Dockerfile.testextendspgvector/pgvector:pg15with PostGIS 3 so both extensions coexist in the test DBosm-db-helpers.tsseedshiking_ways/hiking_relationsrows with real PostGIS geometry via raw SQLTo test the POC