Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
fcca6fb
docs: add modular distribution builder design
robfrank Jan 19, 2026
08cca6e
feat: add base assembly descriptor for modular distribution
robfrank Jan 19, 2026
a615374
feat: add base assembly execution to Maven POM
robfrank Jan 19, 2026
45dc1eb
feat: add SHA-256 checksum generation for distributions
robfrank Jan 19, 2026
e81e4e5
feat: create modular builder script foundation
robfrank Jan 19, 2026
032388c
feat: add help and argument parsing to builder script
robfrank Jan 19, 2026
b0f44c8
feat: add logging and error handling to builder script
robfrank Jan 19, 2026
b67b901
feat: add prerequisites validation to builder script
robfrank Jan 19, 2026
d26ec1e
feat: add version validation and defaults to builder script
robfrank Jan 20, 2026
ab7f837
feat: add interactive module selection to builder script
robfrank Jan 20, 2026
a4e2796
feat: add download and checksum verification functions
robfrank Jan 20, 2026
3e798dd
feat: add base distribution download to builder script
robfrank Jan 20, 2026
866b788
feat: add optional module download from Maven Central
robfrank Jan 20, 2026
8bf3bf9
feat: add archive creation to builder script
robfrank Jan 20, 2026
6f024fd
feat: add Docker image generation to builder script
robfrank Jan 20, 2026
87895f9
fix: add validation for conflicting Docker flags
robfrank Jan 20, 2026
8dc1c12
feat: add build summary to builder script
robfrank Jan 20, 2026
b3907ae
docs: add comprehensive README for builder script
robfrank Jan 20, 2026
6116934
build: add release preparation script for builder
robfrank Jan 20, 2026
4093028
test: add local testing script for builder
robfrank Jan 20, 2026
02ccb95
docs: add modular builder information to CLAUDE.md
robfrank Jan 20, 2026
f5a88ec
docs: add modular builder user guide
robfrank Jan 20, 2026
abeff69
docs: add feature request for modular distribution builder
robfrank Jan 20, 2026
8dc57eb
add implementation plan
robfrank Jan 20, 2026
f075fd3
fix: clarify help text for non-interactive mode and argument format
robfrank Jan 20, 2026
329e098
fix: make test script version dynamic from pom.xml
robfrank Jan 20, 2026
e8e34e8
docs: fix Maven Central URL construction in plan
robfrank Jan 20, 2026
f0c37e3
docs: add implementation plan for local filesystem mode
robfrank Jan 20, 2026
483b549
feat: add --local-repo flag for local filesystem mode
robfrank Jan 20, 2026
5f2a641
feat: add validation for local repository path
robfrank Jan 20, 2026
80e7a79
feat: add find_local_jar helper function
robfrank Jan 20, 2026
219e692
feat: implement local filesystem mode for modules
robfrank Jan 20, 2026
1cdce66
docs: add local repository mode documentation and tests
robfrank Jan 20, 2026
561a82c
fix: prevent silent failure when module not found in local repo
robfrank Jan 20, 2026
57ca5ba
feat: add --local-base flag for local base distribution
robfrank Jan 20, 2026
dc676af
fix: correct extracted directory name (remove -base suffix)
robfrank Jan 20, 2026
bdaeef3
docs: update documentation with --local-base flag examples
robfrank Jan 20, 2026
8f82590
ci: add builder-tests job to CI workflow
robfrank Jan 20, 2026
70bd55b
add skipTests
robfrank Jan 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/mvn-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -202,23 +202,51 @@
**/jacoco*.xml
retention-days: 1

builder-tests:
runs-on: ubuntu-latest
needs: build-and-package
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1

- name: Set up JDK 21
uses: actions/setup-java@f2beeb24e141e01a676f977032f5a29d81c9e27e # v5.1.0
with:
distribution: "temurin"
java-version: 21
cache: "maven"

- name: Restore Maven artifacts
uses: actions/cache/restore@8b402f58fbc84540c8b491a91e594a4576fec3d7 # v5.0.2
with:
path: ~/.m2/repository
key: maven-repo-${{ github.run_id }}-${{ github.run_attempt }}

- name: Build packages for Builder Tests
run: ./mvnw clean package -DskipTests --batch-mode --errors --show-version

- name: Run Builder Tests
working-directory: package
run: ./test-builder-local.sh
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

java-e2e-tests:
runs-on: ubuntu-latest
needs: build-and-package
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1

- name: Set up JDK 21
uses: actions/setup-java@f2beeb24e141e01a676f977032f5a29d81c9e27e # v5.1.0
with:
distribution: "temurin"
java-version: 21
cache: "maven"

- name: Restore Maven artifacts
uses: actions/cache/restore@8b402f58fbc84540c8b491a91e594a4576fec3d7 # v5.0.2
with:
path: ~/.m2/repository

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
key: maven-repo-${{ github.run_id }}-${{ github.run_attempt }}

- name: Restore Docker image
Expand Down
32 changes: 32 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,37 @@
- **Start server**: Use packaged scripts in `package/src/main/scripts/server.sh` (Unix) or `server.bat` (Windows)
- **Console**: Use `package/src/main/scripts/console.sh` or `console.bat`

### Distribution Builder

The modular distribution builder (`package/arcadedb-builder.sh`) creates custom ArcadeDB distributions:

Check notice on line 29 in CLAUDE.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

CLAUDE.md#L29

Expected: 80; Actual: 103

**Production builds** (download from releases):
```bash

Check notice on line 32 in CLAUDE.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

CLAUDE.md#L32

Fenced code blocks should be surrounded by blank lines
cd package
./arcadedb-builder.sh --version=26.1.0 --modules=gremlin,studio
```

**Development builds** (use local Maven repository):
```bash
# Build modules first
mvn clean install -DskipTests

# Create distribution with local modules
cd package
VERSION=$(mvn -f ../pom.xml help:evaluate -Dexpression=project.version -q -DforceStdout)
./arcadedb-builder.sh \
--version=$VERSION \
--modules=console,gremlin,studio \
--local-repo \
--skip-docker
```

**Testing the builder**:
```bash
cd package
./test-builder-local.sh
```

### Testing Commands
- **Run specific test class**: `mvn test -Dtest=ClassName`
- **Run tests with specific pattern**: `mvn test -Dtest="*Pattern*"`
Expand Down Expand Up @@ -143,3 +174,4 @@
- **Performance**: Always consider memory and CPU impact of changes
- **Compatibility**: Maintain backward compatibility for API changes
- **Licensing**: All code must comply with Apache 2.0 license
- **Modular Builder**: Script to create custom distributions with selected modules (see `package/README-BUILDER.md`)

Check notice on line 177 in CLAUDE.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

CLAUDE.md#L177

Expected: 80; Actual: 116
251 changes: 251 additions & 0 deletions docs/feature-request-modular-builder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
# Feature Request: Modular Distribution Builder

## Summary

Add a modular distribution builder that allows users to create custom ArcadeDB packages containing only the modules they need, resulting in smaller distributions, reduced dependencies, and simplified deployments.

## Problem Statement

ArcadeDB currently ships three fixed distributions:
- **Full** (~195MB): All modules included
- **Minimal** (~100MB): Excludes gremlin, redisw, mongodbw, graphql
- **Headless** (~90MB): Minimal without studio

Many users need custom combinations (e.g., only PostgreSQL protocol, or only Gremlin + Studio) but cannot create them without building from source and modifying Maven configuration. This creates unnecessary complexity and larger-than-needed deployments.

## Proposed Solution

Create a standalone builder script (`arcadedb-builder.sh`) that:
1. Downloads a minimal base distribution (engine, server, network + dependencies) from GitHub releases
2. Adds user-selected optional modules from Maven Central
3. Generates zip, tar.gz, and optionally Docker images

### Available Optional Modules

- `console` - Interactive database console
- `gremlin` - Apache Tinkerpop Gremlin support
- `studio` - Web-based administration interface
- `redisw` - Redis wire protocol compatibility
- `mongodbw` - MongoDB wire protocol compatibility
- `postgresw` - PostgreSQL wire protocol compatibility
- `grpcw` - gRPC wire protocol support
- `graphql` - GraphQL API support
- `metrics` - Prometheus metrics integration

## User Experience

### Interactive Mode
```bash
./arcadedb-builder.sh
```
Prompts for version and modules.

### CLI Mode
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw,metrics
```

### Example Use Cases

**PostgreSQL-only deployment:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw
```
Result: ~60MB distribution (vs 195MB full)

**Development environment:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=console,gremlin,studio
```

**Production with monitoring:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw,metrics --skip-docker
```

## Benefits

### For Users
- **Smaller distributions**: 50-70% size reduction for minimal builds
- **Reduced attack surface**: Only include needed protocols
- **Simplified deployments**: No unnecessary dependencies
- **Faster downloads**: Smaller packages download faster
- **Custom Docker images**: Build images with only required modules

### For DevOps
- **CI/CD friendly**: Fully automated via CLI flags
- **Reproducible builds**: Version and module selection in scripts
- **Security compliance**: Minimal installations for regulated environments
- **Bandwidth savings**: Smaller distributions for edge deployments

## Technical Implementation

### Architecture

**Base Distribution Approach:**
- Maven builds a `base` distribution containing core modules + transitive dependencies
- Published to GitHub releases as `arcadedb-{version}-base.tar.gz` (with SHA-256 checksum)
- Base is ~52MB (vs 195MB full distribution)

**Builder Script:**
- Bash 3.2+ compatible (works on macOS and Linux)
- Downloads base from GitHub releases
- Downloads optional modules from Maven Central
- Verifies checksums (SHA-256 for base, SHA-1 for modules)
- Creates zip, tar.gz archives
- Optionally generates Docker images

**Security:**
- URL validation (http/https only)
- Path traversal protection
- Checksum verification before extraction
- Tar bomb prevention
- Download timeouts

### Command-Line Options

**Required (non-interactive):**
- `--version=X.Y.Z` - ArcadeDB version to build

**Optional:**
- `--modules=mod1,mod2,...` - Comma-separated modules
- `--output-name=NAME` - Custom output name
- `--output-dir=PATH` - Output directory
- `--docker-tag=TAG` - Docker image tag
- `--skip-docker` - Skip Docker image generation
- `--dockerfile-only` - Generate Dockerfile without building
- `--keep-temp` - Keep temporary files
- `--dry-run` - Show what would be done
- `-v, --verbose` - Verbose output
- `-q, --quiet` - Quiet mode
- `-h, --help` - Help message

## Implementation Deliverables

### Maven Build Changes
1. **New assembly descriptor** (`package/src/main/assembly/base.xml`):
- Includes engine, server, network modules
- Includes all transitive dependencies
- Excludes all 9 optional modules

2. **Updated `package/pom.xml`**:
- New `base` assembly execution
- SHA-256 checksum generation for all distributions

### Builder Script
- **File**: `package/arcadedb-builder.sh` (~900 lines)
- **Features**: Interactive mode, CLI mode, dry-run, Docker support
- **Error handling**: Strict mode, validation, clear error messages
- **Platform support**: macOS, Linux, Windows WSL

### Documentation
- **User README**: `package/README-BUILDER.md` - Usage guide with examples
- **Developer guide**: `docs/modular-builder-guide.md` - Architecture and customization
- **Project docs**: Updated `CLAUDE.md` with builder commands

### Testing & Release Tools
- **Local testing**: `package/test-builder-local.sh` - Test without GitHub/Maven
- **Release prep**: `package/prepare-release.sh` - Prepare artifacts for GitHub releases

## Release Process

### Build Time
```bash
cd package
mvn clean package -DskipTests
```
Generates:
- `arcadedb-{version}-base.tar.gz` + `.sha256`
- `arcadedb-{version}-base.zip` + `.sha256`
- Standard distributions (full, minimal, headless)

### GitHub Release
Upload to releases:
- `arcadedb-builder.sh` - The builder script
- `README-BUILDER.md` - User documentation
- `arcadedb-{version}-base.tar.gz` + checksum
- Optional: `arcadedb-{version}-base.zip` + checksum

### User Download
Users download `arcadedb-builder.sh` and run:
```bash
chmod +x arcadedb-builder.sh
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw
```

## Testing

### Automated Tests
- 4 dry-run tests validating different scenarios
- Version validation (accepts X.Y.Z and X.Y.Z-SNAPSHOT)
- Invalid input rejection
- Help message display

### Manual Testing
- Built and verified base distribution (~52MB)
- Tested with multiple module combinations
- Verified all 9 optional modules download correctly
- Confirmed Docker image generation works

## Backward Compatibility

- **No breaking changes**: Existing distributions (full, minimal, headless) unchanged
- **Optional feature**: Users can continue using pre-built distributions
- **Additive only**: New base distribution and builder script are additions

## Dependencies

### Runtime (for builder script)
- `curl` or `wget` - File downloads
- `tar` - Archive operations
- `zip`/`unzip` - Zip operations
- `sha256sum` or `shasum` - Checksum verification
- `docker` (optional) - Docker image generation

### Build Time (for base distribution)
- Maven 3.6+
- Java 21+
- Existing ArcadeDB build dependencies

## Size Comparison

| Distribution | Size | Reduction |
|--------------|------|-----------|
| Full | 195MB | baseline |
| Minimal | 100MB | 49% smaller |
| Headless | 90MB | 54% smaller |
| **Base** | **52MB** | **73% smaller** |
| PostgreSQL-only* | ~60MB | 69% smaller |
| Gremlin+Studio* | ~110MB | 44% smaller |

*Custom builds (base + selected modules)

## Future Enhancements

Potential future improvements (not in initial scope):
- GPG signature verification
- Local artifact caching
- Custom Maven repository support
- Configuration file support (YAML/JSON) for repeatable builds
- Integration with package managers (Homebrew, apt, yum)
- Web-based configurator

## Implementation Status

✅ **COMPLETE** - All 21 implementation tasks finished:
- Maven build configuration
- Base assembly descriptor
- Builder script (907 lines, fully functional)
- Security hardening
- Docker support
- Comprehensive documentation
- Testing scripts
- Release preparation tools

Ready for integration into ArcadeDB release process.

---

**Issue Labels**: `enhancement`, `distribution`, `build`, `documentation`
**Assignee**: Build/Release team
**Milestone**: Next release (26.2.0 or 27.0.0)
Loading
Loading