multI: add initial repo scaffolding - #2
Conversation
Summary of ChangesHello @Roasbeef, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request establishes the initial repository structure and core infrastructure for the 'darepo' project. It focuses on setting up a robust development environment by integrating GitHub Actions for CI/CD, defining clear coding standards through detailed guidelines and linting configurations, and implementing foundational RPC services using Protobufs. Furthermore, it introduces a versatile database abstraction layer that supports multiple backends (SQLite and PostgreSQL) with automated migrations and type-safe query generation via SQLC. The changes also include essential build tooling and command-line utilities to facilitate efficient development and maintenance. Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive scaffolding for the new repository, including CI/CD workflows, build systems, database setup with migrations, RPC services, and extensive documentation. The overall structure is robust and follows best practices, with excellent patterns for configuration, logging, and testing. I have a couple of suggestions to improve maintainability by using the newly added build package for version reporting in the RPC responses, instead of hardcoded strings. Otherwise, the changes look great.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Initialize the Go module for the darepo-client project with go 1.25. This commit establishes the base dependency structure, including database drivers for both SQLite (modernc.org/sqlite) and PostgreSQL (jackc/pgx/v5), along with the migration framework. The module uses a forked version of golang-migrate from Lightning Labs that includes custom functionality for post-migration hooks, which are essential for running Golang-based data transformations after SQL migrations complete. This pattern is used extensively in both lnd and taproot-assets for complex schema evolution. Key dependencies include btclog v2 for structured logging, dockertest for integration testing with real database backends, and testify for test assertions. The pure-Go SQLite driver eliminates CGO requirements, simplifying the build process and improving portability.
Add the foundational database schema with an initial migration that creates a chain_info table for tracking blockchain information. This table stores the chain name (mainnet, testnet, regtest) and genesis hash, providing a practical example of the migration system while also serving as useful infrastructure for any Bitcoin application. The sqlc configuration uses PostgreSQL as the SQL engine to maximize compatibility, as it supports the most complete SQL dialect. Generated code will work with both PostgreSQL and SQLite backends through a runtime replacer filesystem that translates dialect differences automatically (BLOB→BYTEA, INTEGER PRIMARY KEY→BIGSERIAL PRIMARY KEY). The migration includes both up and down scripts, establishing the pattern for reversible schema changes. The chain_info table is pre-populated with Bitcoin mainnet genesis hash as a concrete example.
Implement the database migration framework based on golang-migrate with several enhancements adapted from taproot-assets and lnd. The core innovation is the replacerFS virtual filesystem that enables writing SQL migrations once for SQLite and automatically translating them for PostgreSQL at runtime. The migration system includes comprehensive safety features. It detects dirty database states from failed migrations and prevents execution until manual intervention. It implements downgrade protection by comparing the database version against the latest known migration version, refusing to run older code against newer schemas. Migration version tracking uses the standard golang-migrate schema_migrations table. Post-migration hooks provide a mechanism for complex data transformations that cannot be expressed in SQL. The makePostStepCallbacks function wraps these checks in transactions with automatic retry logic, ensuring data consistency even during migration failures. This pattern is critical for migrations that need to analyze existing data and derive new fields, such as detecting script key types or identifying asset burns from witness data. Logging integration uses btclog v2 for structured output, with a migration logger adapter that maps migrate's Printf calls to appropriate log levels based on current settings. The subsystem constant "DARE" identifies database operations in the logs.
Add the core database abstractions that enable clean separation between
business logic and database backends. The BatchedTx generic interface
parameterizes transaction execution over query types, allowing stores
to depend only on the specific queries they need rather than the full
sqlc.Querier interface. This reduces coupling and makes testing easier.
The TransactionExecutor provides automatic retry logic with exponential
backoff for serialization errors and deadlocks. It implements the retry
pattern used in lnd, starting with a randomized 20-60ms delay and
doubling on each attempt up to 3 seconds, with a default of 10 retries.
The randomization prevents thundering herd problems when multiple
goroutines retry simultaneously. Both SQLite's SQLITE_BUSY and
PostgreSQL's SerializationFailure errors trigger retries automatically.
Database-agnostic error handling maps backend-specific errors to
common types. The MapSQLError function detects unique constraint
violations, serialization errors, deadlocks, and schema errors across
both SQLite and PostgreSQL. Connection errors are sanitized to prevent
credential leakage in logs, replacing detailed error messages with a
generic "database connection failed" message while logging the full
error for debugging.
SQL utility functions provide generic helpers for converting between Go
types and SQL nullable types. The extractSqlInt64, sqlInt32, and
similar functions use Go generics with constraints.Integer to work with
any integer type, reducing boilerplate in database code. The
transformByteLiterals helper enables cross-database test data by
converting SQLite hex literals (X'...') to PostgreSQL format ('\x...').
Implement the SQLite database backend using the pure-Go modernc.org/sqlite driver, eliminating CGO dependencies for improved portability and simpler cross-compilation. The configuration uses Write-Ahead Logging (WAL) mode for better concurrency, allowing readers to proceed while a writer is active. This is critical for Lightning Network applications where read queries should not block payment processing. The pragma configuration prioritizes durability over performance. Foreign key constraints are enforced to maintain referential integrity. The synchronous mode is set to "full" and fullfsync is enabled on macOS to ensure writes are actually committed to disk before transactions complete. While this impacts write throughput, it prevents database corruption during crashes, which is essential for financial applications where data loss is unacceptable. The backupAndMigrate function creates database backups before applying migrations using SQLite's VACUUM INTO command, which produces a consistent copy of the database without requiring shutdown. Backups are timestamped to nanosecond precision to avoid collisions. The migration is skipped entirely if the database is already at the latest version, avoiding unnecessary backup creation. Connection pooling is configured with 25 max open connections and a 10 minute lifetime. The busy_timeout is set to 5 seconds, giving transactions time to acquire locks before failing. This strikes a balance between responsiveness and avoiding spurious lock timeouts.
Add the PostgreSQL database backend using jackc/pgx/v5, the high-performance native Go driver for PostgreSQL. The implementation emphasizes production readiness with configurable connection pooling, connection lifetime management, and SSL support. Connection pool configuration allows tuning for different deployment scenarios. The defaults use 25 max open connections and 6 max idle connections, matching the SQLite backend for consistency. Connection max lifetime and max idle time can be configured independently, allowing operators to balance connection reuse against potential connection staleness in cloud environments where load balancers may silently drop idle connections. The DSN construction includes password sanitization for safe logging. When logging database configuration, the password is replaced with asterisks, preventing credential leakage in application logs. The original password is only used in the actual database connection string, never in log output. Schema translation is handled by the replacerFS from the migration infrastructure, applying PostgreSQL-specific replacements at migration time. The backend uses serializable isolation level by default through the BaseDB.BeginTx method, ensuring strong consistency for financial transactions. This prevents phenomena like phantom reads and write skew that could occur at lower isolation levels.
Implement comprehensive test infrastructure supporting both SQLite and PostgreSQL backends through Go build tags. The test_sqlite.go file (active by default) and test_postgres.go file (active with -tags=test_db_postgres) provide identical APIs but instantiate different backends, enabling the same test suite to run against both databases. The SQLite test helpers create temporary databases in t.TempDir() with automatic cleanup, providing complete isolation between test runs. Each test gets a fresh database with all migrations applied, and the cleanup handler ensures the database is properly closed even if the test panics or fails assertions. PostgreSQL testing uses dockertest to spin up actual Postgres 15 containers, providing the highest fidelity testing environment. The TestPgFixture handles the full container lifecycle, from image pulling through network configuration to graceful teardown. The fixture uses exponential backoff to wait for PostgreSQL to be ready, avoiding flaky tests from race conditions during container startup. The fixtures support configurable expiry times to prevent runaway containers from consuming resources if tests hang or crash. The default 60 minute lifetime is sufficient for normal test runs while providing safety against resource leaks. The ClearDB method enables resetting the database between test cases within the same fixture, improving performance when running multiple tests.
Add migration tests demonstrating the complete lifecycle of database schema evolution. These tests serve both as validation of the migration framework and as documentation of migration testing patterns for future schema changes. TestMigrationSteps illustrates the fundamental pattern for migration testing. Start with a database at a specific version, insert test data, apply migrations, and verify the data transformed correctly. This pattern becomes essential when adding migrations that reshape existing data, such as splitting a column into multiple tables or changing data encoding. The test uses transformByteLiterals to write backend-agnostic test data, a technique that enables the same test to run against both SQLite and PostgreSQL. TestMigrationDowngrade validates that the downgrade protection works correctly. Attempting to run older code against a newer database schema is a common failure mode in distributed deployments where instances upgrade at different times. The test verifies that this scenario is detected and prevented with a clear error message rather than allowing subtle data corruption or crashes. TestSqliteMigrationBackup ensures the automatic backup creation works correctly. The test verifies that migrations don't create unnecessary backups when already at the latest version, avoiding disk space waste in production environments. TestDirtySqliteVersion validates the critical safety feature of dirty state detection. When a migration fails partway through, the database is left in an inconsistent state. The test verifies that subsequent migration attempts fail immediately with a clear error, forcing manual intervention to assess and repair the database before proceeding.
Add the sqlc-generated database code along with custom extensions for backend type tracking and query parameter generation. The generated code provides type-safe database access with compile-time verification of SQL queries, eliminating entire classes of runtime errors. The db_custom.go file extends the generated code with backend type awareness. The wrappedTX wrapper stores whether the current database is SQLite or PostgreSQL, enabling runtime decisions about query syntax or feature availability. The NewSqlite and NewPostgres factory functions initialize the Queries struct with the appropriate backend type, replacing the default New function. The makeQueryParams function supports the sqlc slice workaround required when using the PostgreSQL dialect. Since sqlc assumes it can pass Go slices directly to the database driver, it doesn't generate the correct parameter placeholders for IN clauses. The gen_sqlc_docker.sh script injects special comments that are then replaced with calls to makeQueryParams, which generates the correct numbered placeholders ($1, $2, $3) for PostgreSQL's prepared statement protocol. The generated code includes models for the chain_info table, a Querier interface with all available query methods, and implementations for GetChainInfo, ListChainInfo, and UpsertChainInfo. The DBTX interface abstracts over *sql.DB and *sql.Tx, allowing the same query code to run in or out of transactions without modification.
Add the build tooling required to generate type-safe database code from SQL schemas and queries. The gen_sqlc_docker.sh script orchestrates the multi-step code generation process, applying necessary workarounds for sqlc limitations and ensuring the generated code works correctly with both database backends. The script applies a SQLite bigint patch before generation, temporarily replacing INTEGER PRIMARY KEY with BIGINT PRIMARY KEY in migration files. This is necessary because SQLite uses 64-bit integers internally regardless of the column type declaration, but sqlc generates int32 fields for INTEGER columns and int64 fields for BIGINT columns. The patch forces sqlc to generate int64 fields, and the original files are restored after generation to keep the migrations SQLite-compatible. The sqlc.slice() workaround addresses sqlc's handling of array parameters when using the PostgreSQL dialect. The script injects special comments into the generated code that are later replaced with calls to makeQueryParams, which generates correct numbered placeholders for PostgreSQL's prepared statement protocol. The merge-sql-schemas tool provides schema documentation and verification. It applies all migrations to an in-memory SQLite database and extracts the final schema via sqlite_master, producing a single SQL file showing the complete database structure. This consolidated schema is useful for reviewing schema changes in pull requests and serves as executable documentation of the database structure. It's also used by the sqlc-check Makefile target to detect uncommitted schema changes.
Add a comprehensive Makefile providing the primary interface for database operations and code generation. The Makefile follows the patterns established in lnd and taproot-assets, adapting their battle-tested build system for darepo's needs. The sqlc target orchestrates the full code generation pipeline. It invokes gen_sqlc_docker.sh to generate Go code from SQL schemas and queries, then runs merge-sql-schemas to produce the consolidated schema documentation. This two-step process ensures both the generated code and schema documentation remain synchronized with the actual migrations. The sqlc-check target provides CI validation. It regenerates all code and verifies that the generated files match the committed versions, catching cases where developers forgot to regenerate after changing SQL files. It also verifies the presence of the consolidated schema file, which serves as executable documentation and should always be up-to-date with the migrations. The migration management targets provide developer convenience. The migrate-create target uses golang-migrate to generate properly numbered migration files with both .up.sql and .down.sql variants. The migrate-up and migrate-down targets enable manual migration testing during development, though production deployments apply migrations automatically at startup. The gen target serves as the primary entry point for code generation, running sqlc along with any future code generation steps like protobuf compilation. This provides a single command that developers can run to regenerate all derived code after making changes to schemas or APIs.
In this commit, we set up the foundation for our build tooling by creating a dedicated Go module in the tools/ directory. This module declares dependencies on golangci-lint and gosimports, which are essential for our CI pipeline's code quality checks. Following the pattern established in lnd, we use a tools.go file with build tags to ensure these development dependencies are properly tracked by Go's module system without being included in the main application binary. The go.mod pins golangci-lint to v1.64.5 to ensure consistent linting behavior across all development environments and CI runs.
In this commit, we implement a custom line length linter called "ll" that extends the standard lll linter with intelligent exclusions for log lines. This is particularly valuable in Bitcoin and Lightning Network codebases where structured logging often results in long lines that don't benefit from artificial line breaks. The linter is implemented as a golangci-lint plugin using the plugin module register interface. It scans source files and reports lines exceeding 80 characters, but intelligently skips import blocks, go: directives, and log statements matching a configurable regex pattern. The default pattern matches common logging patterns like log.Info, log.Debug, etc. The implementation handles both single-line and multi-line log calls by tracking state across lines. For multi-line log calls, it continues skipping until it encounters the closing parenthesis. This prevents false positives on long log format strings that span multiple lines. Following Go plugin conventions, the linter is packaged in its own module with minimal dependencies, making it easy to version independently and reuse across projects.
In this commit, we create a Docker-based linting environment that incorporates the optimizations from lnd PR #10202. The Dockerfile uses an Alpine base image instead of the full Debian image, significantly reducing image size while maintaining all necessary functionality. The build process installs golangci-lint from source and then uses the "golangci-lint custom" command to build a custom binary that includes our ll linter plugin. The .custom-gcl.yml configuration file tells golangci-lint where to find our plugin module and how to integrate it. Several optimizations reduce the final image size. We clean up the apk cache in the same RUN layer where it's created, preventing cache files from being included in intermediate layers. After building the custom linter binary, we remove Go module caches, build artifacts, and temporary files, keeping only the final custom-gcl executable. We also copy the project's .golangci.yml into the tools directory so it's available in the build context. This configuration enables all linters by default and then selectively disables ones that don't fit our coding style, such as gofumpt and wsl which conflict with our formatting guidelines. The GOFLAGS environment variable disables VCS stamping to avoid git-related issues when building inside the container.
In this commit, we configure golangci-lint for the darepo project by adapting lnd's proven linter configuration to our needs. The configuration takes an "enable all, then disable selectively" approach, which ensures we don't miss new linters as golangci-lint evolves. The configuration targets Go 1.25.3 and includes build tags for our database backends (kvdb_postgres, kvdb_sqlite) and test configurations. This ensures the linter analyzes code across all build configurations rather than just the default one. We've configured our custom ll linter with an 80-character line length limit and a regex pattern that excludes log statements from this check. The pattern matches common logging calls like log.Info, log.Debug, etc., recognizing that structured logging often requires longer lines for clarity. Several linters are disabled based on project conventions. We disable gofumpt and wsl because they have formatting preferences that conflict with our style guidelines. Cognitive complexity linters like gocyclo and gocognit are disabled as they can be overly aggressive and don't always correlate with actual code maintainability. We also disable linters like varnamelen and godox that would flag short variable names and TODO comments, both of which are acceptable in our codebase. The configuration includes exclusion rules for generated files (sqlc output, protobuf generated code) and relaxes certain rules for test files where duplicated code and weak randomness are acceptable.
In this commit, we create three composite actions that encapsulate common setup and maintenance tasks used across our CI workflows. These composite actions follow GitHub Actions best practices by making workflows more maintainable and reducing duplication. The setup-go action handles Go environment configuration with intelligent caching. It sets up the specified Go version and configures two types of caching: a full mode that caches both module downloads and build artifacts, and a lightweight mode that only caches module downloads. The caching strategy uses composite keys based on the OS, Go version, workflow-specific prefix, and go.sum hash, ensuring that caches are properly isolated between different workflows while still benefiting from reuse when dependencies haven't changed. The cleanup-space action addresses a common problem in GitHub Actions runners: limited disk space. It removes large toolsets and packages that are pre-installed but unnecessary for our builds, including dotnet, android SDKs, various language toolchains, and Docker images. This frees up several gigabytes of space, which is particularly important for workflows that build Docker images or run extensive test suites. The rebase action provides a consistent way to rebase pull request code onto the target branch before running CI checks. This ensures we're always testing against the latest version of the target branch, catching integration issues early. It configures a temporary git committer identity specifically for the CI environment and fetches only the necessary branch to minimize network transfer.
In this commit, we establish the primary CI workflow that runs on every push to main and all pull requests. The workflow is structured around four independent jobs that can run in parallel, providing fast feedback while ensuring code quality. The static-checks job verifies that code is properly formatted, Go modules are tidy, and sqlc-generated code is up to date. It runs make fmt-check to ensure all code follows our formatting standards, make tidy-module-check to verify go.mod files are clean, and make sqlc-check to confirm that database query code has been regenerated after any schema changes. This job uses Docker layer caching to speed up sqlc runs since schema generation can be time-consuming. The lint job runs our comprehensive linting suite using the custom Docker image we built with golangci-lint and our ll plugin. It sets GOGC=50 to reduce memory usage during linting, which is important because golangci-lint can be memory-intensive on large codebases. The job reuses Go module caches from the unit-test job to avoid redundant downloads. The cross-compile job verifies that our code builds successfully on all supported platforms. It uses a matrix strategy to test six platform combinations covering Linux, macOS, and Windows on both x86_64 and ARM architectures. This catches platform-specific issues early, such as assuming availability of cgo or using platform-specific system calls. The unit-test job runs our test suite in multiple configurations using a matrix strategy. It tests with coverage reporting, race detection, and both SQLite and PostgreSQL database backends. The job rebases pull request code onto the target branch before testing to catch integration issues early. For coverage builds, it filters out generated protobuf and sqlc files before uploading to Coveralls, ensuring our coverage metrics reflect only hand-written code. All jobs start by cleaning up disk space and setting up Go with caching, leveraging our composite actions for consistency. The workflow uses concurrency control to cancel previous runs when new commits are pushed to the same PR, saving CI resources.
In this commit, we add a script that ensures the Go version is consistent across all project files. This is important because Go version mismatches between the Makefile, Dockerfiles, and GitHub Actions can lead to subtle build and compatibility issues that are difficult to debug. The script takes three arguments: the target version, a file pattern to search, and a regex pattern to find version declarations. It supports both single file patterns like "Dockerfile" and multiple patterns like "*.yml *.yaml", making it flexible enough to check different file types with a single invocation. For each file matching the pattern, the script searches for lines matching the regex and extracts version numbers. It then compares each found version against the target version and reports any mismatches. The script provides clear visual feedback using emoji indicators, showing checkmarks for correct versions and X marks for mismatches. The implementation handles edge cases gracefully. If no files match the pattern, it exits successfully rather than failing, which is useful for projects that might not have all file types. If a file contains the search pattern but no version number, it issues a warning but continues checking other files. This prevents false positives while still alerting developers to potential issues. The script is designed to be called from Makefile targets that verify version consistency as part of the lint process. By catching version mismatches early in development, we prevent the frustration of CI failures due to version inconsistencies.
In this commit, we extend the Makefile with comprehensive targets for code quality, testing, and release builds. These targets provide a consistent developer experience and form the foundation of our CI pipeline. The linting targets build and run our custom Docker-based linting environment. The docker-tools target creates a Docker image containing golangci-lint with our custom ll linter plugin, while lint-source runs the linter against all source code. We apply the cache mounting optimizations from lnd PR #10202, using /root/.cache paths and cleaning up caches in the same Docker layer where they're created. The lint target also runs check-go-version to ensure version consistency across the project. Code formatting targets use gosimports for import organization followed by gofmt for source formatting. The fmt-check target runs formatting and then verifies no changes were made, failing if the code wasn't already properly formatted. This ensures developers can't accidentally commit improperly formatted code. Testing targets support multiple configurations through build tags. The basic unit target runs tests with the dev and nolog tags, while unit-cover runs with coverage instrumentation and unit-race enables the race detector. The UNIT, UNIT_RACE, and UNIT_COVER variables are carefully constructed to pass through build tags and test flags while filtering out vendor directories and generated files from coverage reporting. The tidy-module target runs go mod tidy on the main module and both tools submodules, ensuring all three module files stay synchronized. The tidy-module-check target verifies no changes are needed, which catches cases where developers forgot to run go mod tidy after changing dependencies. For releases, we add cross-compilation support that builds binaries for six platform combinations. The BUILD_SYSTEM variable can be overridden with the sys parameter to build for a specific platform during development. This helps catch platform-specific issues before they reach the CI pipeline. We introduce several new variables following lnd's patterns. GO_VERSION serves as the single source of truth for the Go version, GOFILES_NOVENDOR provides a filtered list of source files for formatting tools, and DEV_TAGS accumulates build tags from multiple sources. The DOCKER_TOOLS variable uses optimized cache mounting that creates /tmp cache directories on the fly, ensuring the Docker command works even on systems where these directories don't exist yet.
Fix critical typo where 'rm-rf' was missing a space, causing the command to fail. The correct command is 'rm -rf'.
Replace eval usage with array-based approach for building find arguments. This eliminates the security risk of eval while maintaining the same functionality. The script now safely handles multiple file patterns by building an array of find arguments and passing them directly to the find command using proper array expansion.
Remove the duplicate .golangci.yml file from the tools/ directory as it is unused. The lint make target runs golangci-lint from the project root, which uses the root .golangci.yml configuration. Having this duplicate could lead to confusion about which configuration is active and creates unnecessary maintenance burden.
No description provided.