Skip to content
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
fcca6fb
docs: add modular distribution builder design
robfrank Jan 19, 2026
08cca6e
feat: add base assembly descriptor for modular distribution
robfrank Jan 19, 2026
a615374
feat: add base assembly execution to Maven POM
robfrank Jan 19, 2026
45dc1eb
feat: add SHA-256 checksum generation for distributions
robfrank Jan 19, 2026
e81e4e5
feat: create modular builder script foundation
robfrank Jan 19, 2026
032388c
feat: add help and argument parsing to builder script
robfrank Jan 19, 2026
b0f44c8
feat: add logging and error handling to builder script
robfrank Jan 19, 2026
b67b901
feat: add prerequisites validation to builder script
robfrank Jan 19, 2026
d26ec1e
feat: add version validation and defaults to builder script
robfrank Jan 20, 2026
ab7f837
feat: add interactive module selection to builder script
robfrank Jan 20, 2026
a4e2796
feat: add download and checksum verification functions
robfrank Jan 20, 2026
3e798dd
feat: add base distribution download to builder script
robfrank Jan 20, 2026
866b788
feat: add optional module download from Maven Central
robfrank Jan 20, 2026
8bf3bf9
feat: add archive creation to builder script
robfrank Jan 20, 2026
6f024fd
feat: add Docker image generation to builder script
robfrank Jan 20, 2026
87895f9
fix: add validation for conflicting Docker flags
robfrank Jan 20, 2026
8dc1c12
feat: add build summary to builder script
robfrank Jan 20, 2026
b3907ae
docs: add comprehensive README for builder script
robfrank Jan 20, 2026
6116934
build: add release preparation script for builder
robfrank Jan 20, 2026
4093028
test: add local testing script for builder
robfrank Jan 20, 2026
02ccb95
docs: add modular builder information to CLAUDE.md
robfrank Jan 20, 2026
f5a88ec
docs: add modular builder user guide
robfrank Jan 20, 2026
abeff69
docs: add feature request for modular distribution builder
robfrank Jan 20, 2026
8dc57eb
add implementation plan
robfrank Jan 20, 2026
f075fd3
fix: clarify help text for non-interactive mode and argument format
robfrank Jan 20, 2026
329e098
fix: make test script version dynamic from pom.xml
robfrank Jan 20, 2026
e8e34e8
docs: fix Maven Central URL construction in plan
robfrank Jan 20, 2026
f0c37e3
docs: add implementation plan for local filesystem mode
robfrank Jan 20, 2026
483b549
feat: add --local-repo flag for local filesystem mode
robfrank Jan 20, 2026
5f2a641
feat: add validation for local repository path
robfrank Jan 20, 2026
80e7a79
feat: add find_local_jar helper function
robfrank Jan 20, 2026
219e692
feat: implement local filesystem mode for modules
robfrank Jan 20, 2026
1cdce66
docs: add local repository mode documentation and tests
robfrank Jan 20, 2026
561a82c
fix: prevent silent failure when module not found in local repo
robfrank Jan 20, 2026
57ca5ba
feat: add --local-base flag for local base distribution
robfrank Jan 20, 2026
dc676af
fix: correct extracted directory name (remove -base suffix)
robfrank Jan 20, 2026
bdaeef3
docs: update documentation with --local-base flag examples
robfrank Jan 20, 2026
8f82590
ci: add builder-tests job to CI workflow
robfrank Jan 20, 2026
70bd55b
add skipTests
robfrank Jan 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@
- **Start server**: Use packaged scripts in `package/src/main/scripts/server.sh` (Unix) or `server.bat` (Windows)
- **Console**: Use `package/src/main/scripts/console.sh` or `console.bat`

### Modular Distribution Builder

Check notice on line 27 in CLAUDE.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

CLAUDE.md#L27

Expected: 1; Actual: 0; Below
- **Build custom distribution**: `package/arcadedb-builder.sh --version=X.Y.Z --modules=mod1,mod2`

Check notice on line 28 in CLAUDE.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

CLAUDE.md#L28

Lists should be surrounded by blank lines
- **Interactive mode**: `package/arcadedb-builder.sh`
- **See options**: `package/arcadedb-builder.sh --help`
- **Local testing**: `package/test-builder-local.sh`

### Testing Commands
- **Run specific test class**: `mvn test -Dtest=ClassName`
- **Run tests with specific pattern**: `mvn test -Dtest="*Pattern*"`
Expand Down Expand Up @@ -143,3 +149,4 @@
- **Performance**: Always consider memory and CPU impact of changes
- **Compatibility**: Maintain backward compatibility for API changes
- **Licensing**: All code must comply with Apache 2.0 license
- **Modular Builder**: Script to create custom distributions with selected modules (see `package/README-BUILDER.md`)

Check notice on line 152 in CLAUDE.md

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

CLAUDE.md#L152

Expected: 80; Actual: 116
251 changes: 251 additions & 0 deletions docs/feature-request-modular-builder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
# Feature Request: Modular Distribution Builder

## Summary

Add a modular distribution builder that allows users to create custom ArcadeDB packages containing only the modules they need, resulting in smaller distributions, reduced dependencies, and simplified deployments.

## Problem Statement

ArcadeDB currently ships three fixed distributions:
- **Full** (~195MB): All modules included
- **Minimal** (~100MB): Excludes gremlin, redisw, mongodbw, graphql
- **Headless** (~90MB): Minimal without studio

Many users need custom combinations (e.g., only PostgreSQL protocol, or only Gremlin + Studio) but cannot create them without building from source and modifying Maven configuration. This creates unnecessary complexity and larger-than-needed deployments.

## Proposed Solution

Create a standalone builder script (`arcadedb-builder.sh`) that:
1. Downloads a minimal base distribution (engine, server, network + dependencies) from GitHub releases
2. Adds user-selected optional modules from Maven Central
3. Generates zip, tar.gz, and optionally Docker images

### Available Optional Modules

- `console` - Interactive database console
- `gremlin` - Apache Tinkerpop Gremlin support
- `studio` - Web-based administration interface
- `redisw` - Redis wire protocol compatibility
- `mongodbw` - MongoDB wire protocol compatibility
- `postgresw` - PostgreSQL wire protocol compatibility
- `grpcw` - gRPC wire protocol support
- `graphql` - GraphQL API support
- `metrics` - Prometheus metrics integration

## User Experience

### Interactive Mode
```bash
./arcadedb-builder.sh
```
Prompts for version and modules.

### CLI Mode
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw,metrics
```

### Example Use Cases

**PostgreSQL-only deployment:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw
```
Result: ~60MB distribution (vs 195MB full)

**Development environment:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=console,gremlin,studio
```

**Production with monitoring:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw,metrics --skip-docker
```

## Benefits

### For Users
- **Smaller distributions**: 50-70% size reduction for minimal builds
- **Reduced attack surface**: Only include needed protocols
- **Simplified deployments**: No unnecessary dependencies
- **Faster downloads**: Smaller packages download faster
- **Custom Docker images**: Build images with only required modules

### For DevOps
- **CI/CD friendly**: Fully automated via CLI flags
- **Reproducible builds**: Version and module selection in scripts
- **Security compliance**: Minimal installations for regulated environments
- **Bandwidth savings**: Smaller distributions for edge deployments

## Technical Implementation

### Architecture

**Base Distribution Approach:**
- Maven builds a `base` distribution containing core modules + transitive dependencies
- Published to GitHub releases as `arcadedb-{version}-base.tar.gz` (with SHA-256 checksum)
- Base is ~52MB (vs 195MB full distribution)

**Builder Script:**
- Bash 3.2+ compatible (works on macOS and Linux)
- Downloads base from GitHub releases
- Downloads optional modules from Maven Central
- Verifies checksums (SHA-256 for base, SHA-1 for modules)
- Creates zip, tar.gz archives
- Optionally generates Docker images

**Security:**
- URL validation (http/https only)
- Path traversal protection
- Checksum verification before extraction
- Tar bomb prevention
- Download timeouts

### Command-Line Options

**Required (non-interactive):**
- `--version=X.Y.Z` - ArcadeDB version to build

**Optional:**
- `--modules=mod1,mod2,...` - Comma-separated modules
- `--output-name=NAME` - Custom output name
- `--output-dir=PATH` - Output directory
- `--docker-tag=TAG` - Docker image tag
- `--skip-docker` - Skip Docker image generation
- `--dockerfile-only` - Generate Dockerfile without building
- `--keep-temp` - Keep temporary files
- `--dry-run` - Show what would be done
- `-v, --verbose` - Verbose output
- `-q, --quiet` - Quiet mode
- `-h, --help` - Help message

## Implementation Deliverables

### Maven Build Changes
1. **New assembly descriptor** (`package/src/main/assembly/base.xml`):
- Includes engine, server, network modules
- Includes all transitive dependencies
- Excludes all 9 optional modules

2. **Updated `package/pom.xml`**:
- New `base` assembly execution
- SHA-256 checksum generation for all distributions

### Builder Script
- **File**: `package/arcadedb-builder.sh` (~900 lines)
- **Features**: Interactive mode, CLI mode, dry-run, Docker support
- **Error handling**: Strict mode, validation, clear error messages
- **Platform support**: macOS, Linux, Windows WSL

### Documentation
- **User README**: `package/README-BUILDER.md` - Usage guide with examples
- **Developer guide**: `docs/modular-builder-guide.md` - Architecture and customization
- **Project docs**: Updated `CLAUDE.md` with builder commands

### Testing & Release Tools
- **Local testing**: `package/test-builder-local.sh` - Test without GitHub/Maven
- **Release prep**: `package/prepare-release.sh` - Prepare artifacts for GitHub releases

## Release Process

### Build Time
```bash
cd package
mvn clean package -DskipTests
```
Generates:
- `arcadedb-{version}-base.tar.gz` + `.sha256`
- `arcadedb-{version}-base.zip` + `.sha256`
- Standard distributions (full, minimal, headless)

### GitHub Release
Upload to releases:
- `arcadedb-builder.sh` - The builder script
- `README-BUILDER.md` - User documentation
- `arcadedb-{version}-base.tar.gz` + checksum
- Optional: `arcadedb-{version}-base.zip` + checksum

### User Download
Users download `arcadedb-builder.sh` and run:
```bash
chmod +x arcadedb-builder.sh
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw
```

## Testing

### Automated Tests
- 4 dry-run tests validating different scenarios
- Version validation (accepts X.Y.Z and X.Y.Z-SNAPSHOT)
- Invalid input rejection
- Help message display

### Manual Testing
- Built and verified base distribution (~52MB)
- Tested with multiple module combinations
- Verified all 9 optional modules download correctly
- Confirmed Docker image generation works

## Backward Compatibility

- **No breaking changes**: Existing distributions (full, minimal, headless) unchanged
- **Optional feature**: Users can continue using pre-built distributions
- **Additive only**: New base distribution and builder script are additions

## Dependencies

### Runtime (for builder script)
- `curl` or `wget` - File downloads
- `tar` - Archive operations
- `zip`/`unzip` - Zip operations
- `sha256sum` or `shasum` - Checksum verification
- `docker` (optional) - Docker image generation

### Build Time (for base distribution)
- Maven 3.6+
- Java 21+
- Existing ArcadeDB build dependencies

## Size Comparison

| Distribution | Size | Reduction |
|--------------|------|-----------|
| Full | 195MB | baseline |
| Minimal | 100MB | 49% smaller |
| Headless | 90MB | 54% smaller |
| **Base** | **52MB** | **73% smaller** |
| PostgreSQL-only* | ~60MB | 69% smaller |
| Gremlin+Studio* | ~110MB | 44% smaller |

*Custom builds (base + selected modules)

## Future Enhancements

Potential future improvements (not in initial scope):
- GPG signature verification
- Local artifact caching
- Custom Maven repository support
- Configuration file support (YAML/JSON) for repeatable builds
- Integration with package managers (Homebrew, apt, yum)
- Web-based configurator

## Implementation Status

✅ **COMPLETE** - All 21 implementation tasks finished:
- Maven build configuration
- Base assembly descriptor
- Builder script (907 lines, fully functional)
- Security hardening
- Docker support
- Comprehensive documentation
- Testing scripts
- Release preparation tools

Ready for integration into ArcadeDB release process.

---

**Issue Labels**: `enhancement`, `distribution`, `build`, `documentation`
**Assignee**: Build/Release team
**Milestone**: Next release (26.2.0 or 27.0.0)
115 changes: 115 additions & 0 deletions docs/modular-builder-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Modular Distribution Builder Guide

## For End Users

### What is the Modular Builder?

The ArcadeDB Modular Distribution Builder allows you to create custom ArcadeDB packages containing only the features you need. This results in smaller distributions, reduced dependencies, and simplified deployments.

### Getting Started

1. Download `arcadedb-builder.sh` from the [GitHub releases page](https://github.com/arcadedata/arcadedb/releases)
2. Make it executable: `chmod +x arcadedb-builder.sh`
3. Run it: `./arcadedb-builder.sh`

### Quick Examples

**Interactive Mode:**
```bash
./arcadedb-builder.sh
# Follow the prompts to select version and modules
```

**PostgreSQL-only Build:**
```bash
./arcadedb-builder.sh --version=26.1.0 --modules=postgresw
```

**Full Custom Build:**
```bash
./arcadedb-builder.sh \
--version=26.1.0 \
--modules=console,gremlin,studio,postgresw \
--output-name=my-arcadedb
```

### Module Selection Guide

Choose modules based on your needs:

- **console**: If you need the interactive command-line tool
- **gremlin**: If you're using Gremlin graph queries
- **studio**: If you want the web-based admin UI
- **postgresw**: If you're connecting via PostgreSQL protocol
- **mongodbw**: If you're connecting via MongoDB protocol
- **redisw**: If you're connecting via Redis protocol
- **grpcw**: If you're using gRPC
- **graphql**: If you're using GraphQL queries
- **metrics**: If you need Prometheus metrics

### Distribution Comparison

| Distribution | Size | Modules Included |
|--------------|------|------------------|
| Full | ~150MB | All modules |
| Minimal | ~100MB | No gremlin, redisw, mongodbw, graphql |
| Headless | ~90MB | Minimal without studio |
| Custom | Varies | Your selection |

## For Developers

### Building from Source

After building ArcadeDB:

```bash
cd package
mvn clean package -DskipTests
./arcadedb-builder.sh --version=26.1.1-SNAPSHOT --modules=gremlin
```

### Testing the Builder

```bash
cd package
./test-builder-local.sh
```

### Preparing a Release

```bash
cd package
./prepare-release.sh 26.1.0
```

This creates release artifacts in `target/release-26.1.0/`:
- `arcadedb-builder.sh`
- `README-BUILDER.md`

Upload these along with the base distribution to GitHub releases.

### Architecture

The builder works in phases:

1. **Download Base**: Gets core modules from GitHub releases
2. **Add Modules**: Downloads optional modules from Maven Central
3. **Verify**: Checks SHA-256 and SHA-1 checksums
4. **Package**: Creates zip and tar.gz archives
5. **Docker**: Optionally builds Docker image

### Adding New Optional Modules

To add a new optional module:

1. Update `package/pom.xml` dependencies
2. Update base.xml to exclude the new module
3. Update `arcadedb-builder.sh`:
- Add to `SHADED_MODULES` or `REGULAR_MODULES`
- Add description to `MODULE_DESCRIPTIONS`
4. Test with local builder
5. Update documentation

## Troubleshooting

See `package/README-BUILDER.md` for detailed troubleshooting guide.
Loading
Loading