-
Notifications
You must be signed in to change notification settings - Fork 0
feat(bff): add PgBouncer pooling, Redis cache with stampede protectio… #5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
dad20be
6aeb102
a0c7a05
5c9b1a4
1544192
c0891ea
562ad09
51b7622
17f201c
8689719
7c843cc
bc75555
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,3 +20,4 @@ reviews: | |
| Verify eventId and tenantId are always set. | ||
| chat: | ||
| auto_reply: true | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,103 @@ | ||||||||||||
| .PHONY: up down restart logs build seed test lint clean help | ||||||||||||
|
|
||||||||||||
| # ============================================ | ||||||||||||
| # GrainGuard — Developer Makefile | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| help: ## Show this help | ||||||||||||
| @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' | ||||||||||||
|
|
||||||||||||
| # ============================================ | ||||||||||||
| # Docker Compose | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| up: ## Start all services | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml up -d | ||||||||||||
|
|
||||||||||||
| down: ## Stop all services | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml down | ||||||||||||
|
|
||||||||||||
| restart: ## Restart all services | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml restart | ||||||||||||
|
|
||||||||||||
| logs: ## Tail logs for all services | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml logs -f | ||||||||||||
|
|
||||||||||||
| logs-gateway: ## Tail gateway logs | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml logs -f gateway | ||||||||||||
|
|
||||||||||||
| logs-bff: ## Tail BFF logs | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml logs -f bff | ||||||||||||
|
|
||||||||||||
| logs-kafka: ## Tail Kafka logs | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml logs -f kafka | ||||||||||||
|
|
||||||||||||
| build: ## Build all Docker images | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml build | ||||||||||||
|
|
||||||||||||
| # ============================================ | ||||||||||||
| # Database | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| seed: ## Seed dev data (Postgres + Elasticsearch) | ||||||||||||
| cd scripts/seed && bash seed-postgres.sh | ||||||||||||
| cd scripts/seed && bash seed-elasticsearch.sh | ||||||||||||
|
|
||||||||||||
| migrate: ## Run database migrations | ||||||||||||
| go run libs/migrate/migrate.go | ||||||||||||
|
|
||||||||||||
| # ============================================ | ||||||||||||
| # Testing | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| test: ## Run all tests | ||||||||||||
| $(MAKE) test-go | ||||||||||||
| $(MAKE) test-react | ||||||||||||
|
|
||||||||||||
| test-go: ## Run Go tests | ||||||||||||
| go test ./... | ||||||||||||
|
|
||||||||||||
| test-react: ## Run React/Node tests | ||||||||||||
| cd apps/dashboard && npm test -- --run | ||||||||||||
| cd apps/bff && npm test -- --run | ||||||||||||
|
|
||||||||||||
| test-load: ## Run k6 load tests | ||||||||||||
| k6 run scripts/load-tests/gateway-load-test.js | ||||||||||||
| k6 run scripts/load-tests/bff-load-test.js | ||||||||||||
|
|
||||||||||||
| # ============================================ | ||||||||||||
| # Linting | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| lint: ## Lint all services | ||||||||||||
| $(MAKE) lint-go | ||||||||||||
| $(MAKE) lint-ts | ||||||||||||
|
|
||||||||||||
| lint-go: ## Lint Go services | ||||||||||||
| golangci-lint run ./... | ||||||||||||
|
|
||||||||||||
| lint-ts: ## Lint TypeScript services | ||||||||||||
| cd apps/gateway && npm run lint | ||||||||||||
| cd apps/bff && npm run lint | ||||||||||||
| cd apps/dashboard && npm run lint | ||||||||||||
|
|
||||||||||||
| # ============================================ | ||||||||||||
| # Cleanup | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| clean: ## Remove all containers and volumes | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml down -v --remove-orphans | ||||||||||||
|
|
||||||||||||
| clean-cache: ## Clear Redis cache | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml exec redis redis-cli FLUSHALL | ||||||||||||
|
|
||||||||||||
|
Comment on lines
+91
to
+93
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: # First, find and verify the Makefile exists
fd -type f -name "Makefile" | head -20Repository: pahuldeepp/GrainGuard- Length of output: 236 🏁 Script executed: # Once located, read the specific lines mentioned (91-93)
cat -n Makefile | sed -n '85,100p'Repository: pahuldeepp/GrainGuard- Length of output: 660 🏁 Script executed: # Check the docker-compose configuration for Redis setup
fd -type f -name "docker-compose.yml" | head -10Repository: pahuldeepp/GrainGuard- Length of output: 236 🏁 Script executed: # Find docker-compose.yml with correct syntax
find . -name "docker-compose.yml" -type fRepository: pahuldeepp/GrainGuard- Length of output: 99 🏁 Script executed: # Look at the docker-compose configuration
cat infra/docker/docker-compose.ymlRepository: pahuldeepp/GrainGuard- Length of output: 14710 🏁 Script executed: # Check if there are any references to Redis DB selection in the codebase
rg "redis|Redis|REDIS" --max-count=20 -iRepository: pahuldeepp/GrainGuard- Length of output: 28172 Use Line 92 executes Fix clean-cache: ## Clear Redis cache
- docker compose -f infra/docker/docker-compose.yml exec redis redis-cli FLUSHALL
+ docker compose -f infra/docker/docker-compose.yml exec redis redis-cli FLUSHDB📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||
| # ============================================ | ||||||||||||
| # Status | ||||||||||||
| # ============================================ | ||||||||||||
|
|
||||||||||||
| ps: ## Show running services | ||||||||||||
| docker compose -f infra/docker/docker-compose.yml ps | ||||||||||||
|
|
||||||||||||
| health: ## Check health of all services | ||||||||||||
| curl -s http://localhost:8086/health | jq | ||||||||||||
| curl -s http://localhost:4000/health | jq | ||||||||||||
|
Comment on lines
+101
to
+103
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: # First, locate and examine the Makefile
find . -name "Makefile" -type f | head -5Repository: pahuldeepp/GrainGuard- Length of output: 76 🏁 Script executed: # Once we have the Makefile, check the health target
sed -n '95,110p' MakefileRepository: pahuldeepp/GrainGuard- Length of output: 332 🌐 Web query:
💡 Result:
Common script pattern: curl -fsS https://example.com/resource # silent, show errors, fail (exit!=0) on 4xx/5xxSources: [1] [2] [3] [4] Add Lines 102–103 use Proposed fix health: ## Check health of all services
- curl -s http://localhost:8086/health | jq
- curl -s http://localhost:4000/health | jq
+ curl -fsS --max-time 5 http://localhost:8086/health | jq
+ curl -fsS --max-time 5 http://localhost:4000/health | jq🤖 Prompt for AI Agents |
||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| # GrainGuard | ||
|
|
||
| > Production-grade, multi-tenant agri-operations SaaS platform. | ||
| > Built as a Staff+/Principal Engineer portfolio reference system. | ||
|
|
||
| [](https://github.com/pahuldeepp/GrainGuard-/actions) | ||
|
|
||
| --- | ||
|
|
||
| ## What is GrainGuard? | ||
|
|
||
| GrainGuard ingests telemetry from grain bins and field devices (temperature, | ||
| humidity, CO2), computes spoilage risk scores, and automates alert workflows. | ||
|
|
||
| It is deliberately architected to demonstrate end-to-end distributed systems | ||
| patterns from DDIA and Staff-level engineering practices. | ||
|
|
||
| --- | ||
|
|
||
| ## Architecture | ||
| ``` | ||
| React/Next.js Dashboard | ||
| ↓ | ||
| GraphQL Gateway (Node) — auth, rate limiting, versioning | ||
| ↓ | ||
| BFF (Node/Apollo) — resolvers, Redis cache, circuit breaker | ||
| ↓ | ||
| ┌─────────────────────────────────────────┐ | ||
| │ Telemetry Ingest (Go) │ | ||
| │ Asset Registry (Go) │ | ||
| │ Risk Engine (Python) │ | ||
| │ Workflow Alerts (Node) │ | ||
| │ Search Indexer (Python) │ | ||
| │ Jobs Worker (Node) │ | ||
| └─────────────────────────────────────────┘ | ||
| ↓ | ||
| Kafka (domain + CDC) — RabbitMQ (tasks) | ||
| ↓ | ||
| ┌─────────────────────────────────────────┐ | ||
| │ Postgres (OLTP + Outbox) │ | ||
| │ Cassandra (Time-series telemetry) │ | ||
| │ Elasticsearch (Search) │ | ||
| │ Redis (Cache + Locks + Rate limiting) │ | ||
| │ Memcached (Volatile dashboard cache) │ | ||
| └─────────────────────────────────────────┘ | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Key Engineering Patterns | ||
|
|
||
| | Pattern | Implementation | | ||
| |---|---| | ||
| | CQRS | Separate write (Postgres) and read (projections) models | | ||
| | Transactional Outbox | Reliable event publishing without dual-write | | ||
| | Saga Orchestration | Device provisioning with compensation | | ||
| | CDC | Debezium + Kafka Connect for Postgres WAL streaming | | ||
| | Exactly-once-ish | `processed_events` unique insert + projection UPSERT | | ||
| | Multi-tenancy | Postgres RLS + tenant-scoped queries | | ||
| | Rate limiting | Redis sliding window — 7-layer architecture | | ||
| | Cache stampede | Redis distributed lock on cache miss | | ||
| | Circuit breaker | BFF → Postgres, CLOSED/OPEN/HALF_OPEN states | | ||
| | gRPC + mTLS | Internal service communication with zero-trust mesh | | ||
|
|
||
| --- | ||
|
|
||
| ## Tech Stack | ||
|
|
||
| **Languages:** Go, TypeScript/Node, Python | ||
| **Frontend:** React, Apollo Client, Tailwind, Storybook | ||
| **Databases:** Postgres, Cassandra, Elasticsearch, Redis, Memcached | ||
| **Messaging:** Kafka (KRaft), RabbitMQ, Debezium | ||
| **Infra:** Docker, Kubernetes, Helm | ||
| **Observability:** OpenTelemetry, Prometheus, Grafana, Loki, Tempo | ||
| **Auth:** Auth0, JWT RS256, RBAC | ||
| **CI/CD:** GitHub Actions | ||
|
|
||
| --- | ||
|
|
||
| ## Getting Started | ||
|
|
||
| ### Prerequisites | ||
| - Docker + Docker Compose | ||
| - Node.js 20+ | ||
| - Go 1.25+ | ||
| - Python 3.12+ | ||
|
|
||
| ### One-command bootstrap | ||
| ```bash | ||
| bash scripts/bootstrap.sh | ||
| ``` | ||
|
|
||
| ### Manual setup | ||
| ```bash | ||
| # Start all infrastructure | ||
| make up | ||
|
|
||
| # Seed dev data | ||
| make seed | ||
|
|
||
| # Run tests | ||
| make test | ||
| ``` | ||
|
|
||
| ### Available commands | ||
| ```bash | ||
| make help | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Services | ||
|
|
||
| | Service | Language | Port | Description | | ||
| |---|---|---|---| | ||
| | gateway | Node | 8086 | GraphQL gateway, auth, rate limiting | | ||
| | bff | Node | 4000 | Apollo GraphQL BFF | | ||
| | dashboard | React | 5173 | Frontend UI | | ||
| | telemetry-service | Go | 50051 | gRPC telemetry ingest | | ||
| | read-model-builder | Go | - | Kafka → Postgres projections | | ||
| | saga-orchestrator | Go | - | Device provisioning saga | | ||
| | cdc-transformer | Go | - | Debezium CDC transformer | | ||
| | search-indexer | Python | - | Elasticsearch indexer | | ||
| | jobs-worker | Node | - | RabbitMQ async jobs | | ||
|
|
||
| --- | ||
|
|
||
| ## Observability | ||
|
|
||
| | Tool | URL | Purpose | | ||
| |---|---|---| | ||
| | Grafana | http://localhost:3000 | Dashboards + alerts | | ||
| | Prometheus | http://localhost:9090 | Metrics | | ||
| | Kibana | http://localhost:5601 | Elasticsearch UI | | ||
| | RabbitMQ UI | http://localhost:15672 | Queue management | | ||
| | Schema Registry | http://localhost:8082 | Protobuf schemas | | ||
|
|
||
| --- | ||
|
|
||
| ## Architecture Decision Records | ||
|
|
||
| 10 ADRs documented in [`/docs/adr`](/docs/adr): | ||
|
|
||
| | ADR | Decision | | ||
| |---|---| | ||
| | ADR-001 | gRPC for internal service communication | | ||
| | ADR-002 | Transactional outbox over dual writes | | ||
| | ADR-003 | SAGA orchestration for device provisioning | | ||
| | ADR-004 | Cassandra TWCS for telemetry time-series | | ||
| | ADR-005 | Postgres RLS for standard-tier tenant isolation | | ||
| | ADR-006 | Redis lock with fencing tokens | | ||
| | ADR-007 | Reject cross-datastore 2PC | | ||
| | ADR-008 | Kafka KRaft over ZooKeeper | | ||
| | ADR-009 | Serializable isolation for write-skew ops | | ||
| | ADR-010 | etcd leases for singleton jobs | | ||
|
|
||
| --- | ||
|
|
||
| ## Load Testing Results | ||
|
|
||
| - **Gateway:** 1,700 events/sec, p95 5.89ms | ||
| - **Test tool:** k6 spike/soak/stress | ||
| - **Results:** [`/docs/load`](/docs/load) | ||
|
|
||
| --- | ||
|
|
||
| ## Project Status | ||
|
|
||
| | Milestone | Status | | ||
| |---|---| | ||
| | R1 — Core correctness loop | 🟡 In Progress | | ||
| | R2 — CDC + Search | ✅ Done | | ||
| | R3 — Massive scale | 🔴 Pending | | ||
| | R4 — 1B hardening | 🔴 Pending | | ||
|
|
||
| --- | ||
|
|
||
| ## License | ||
|
|
||
| Private — portfolio project. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,3 +69,4 @@ export async function closeCassandraClient(): Promise<void> { | |
| client = null; | ||
| } | ||
| } | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -51,3 +51,4 @@ export const search = { | |
| } | ||
| }, | ||
| }; | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: pahuldeepp/GrainGuard-
Length of output: 1021
Add a default
alltarget and complete the.PHONYlist.The
.PHONYdeclaration on line 1 is incomplete, missing 12 targets that are actually defined later in the Makefile:logs-gateway,logs-bff,logs-kafka,migrate,test-go,test-react,test-load,lint-go,lint-ts,clean-cache,ps, andhealth. Additionally, thealltarget is missing entirely, which is a standard Makefile convention. These omissions can cause target/file name collisions and potential CI lint failures.Proposed fix
📝 Committable suggestion
🧰 Tools
🪛 checkmake (0.2.2)
[warning] 1-1: Missing required phony target "all"
(minphony)
🤖 Prompt for AI Agents