|
| 1 | +# Directory Structure Proposal |
| 2 | + |
| 3 | +## 🎯 Overview |
| 4 | + |
| 5 | +This document proposes a clean separation between source code and user data for the Torrust |
| 6 | +Tracker automation project. The proposed structure separates version-controlled application |
| 7 | +logic from user-specific configurations and generated outputs. |
| 8 | + |
| 9 | +This structure enables a single automation tool to manage multiple environments while keeping |
| 10 | +sensitive data separate from the main repository. Users maintain their configurations outside |
| 11 | +the main repository while using the standardized automation tooling. |
| 12 | + |
| 13 | +The design supports diverse deployment scenarios from individual developers testing locally |
| 14 | +to enterprise teams managing multiple production environments. |
| 15 | + |
| 16 | +## 🏗️ Design Principles |
| 17 | + |
| 18 | +1. **Clear Separation of Concerns**: Distinct directories for source code, user inputs, |
| 19 | + and generated outputs |
| 20 | +2. **Environment Isolation**: Each environment has its own configuration space |
| 21 | +3. **Provider Profile Constraints**: One environment uses exactly one provider profile |
| 22 | +4. **Security Boundaries**: Secrets and credentials isolated from source code |
| 23 | + |
| 24 | +## 📁 Proposed Directory Structure |
| 25 | + |
| 26 | +### Root Level Structure |
| 27 | + |
| 28 | +```text |
| 29 | +torrust-tracker-installer/ |
| 30 | +├── README.md # Project documentation |
| 31 | +├── LICENSE # Project license |
| 32 | +├── .gitignore # Excludes data/ |
| 33 | +├── data/ # Data directory (git-ignored) |
| 34 | +│ ├── inputs/ # User data |
| 35 | +│ └── outputs/ # Generated data |
| 36 | +└── src/ # Source code (version controlled) |
| 37 | +``` |
| 38 | + |
| 39 | +### Complete Directory Tree |
| 40 | + |
| 41 | +```text |
| 42 | +torrust-tracker-installer/ |
| 43 | +├── README.md |
| 44 | +├── LICENSE |
| 45 | +├── .gitignore |
| 46 | +├── CHANGELOG.md |
| 47 | +├── CONTRIBUTING.md |
| 48 | +├── data/ # DATA DIRECTORY (not in main repo) |
| 49 | +│ ├── inputs/ # USER DATA |
| 50 | +│ │ └── environments/ # Environment-specific configurations |
| 51 | +│ │ ├── dev-alice/ |
| 52 | +│ │ │ ├── config.yaml # Environment configuration |
| 53 | +│ │ │ ├── provider-profile.yaml # Provider credentials & settings |
| 54 | +│ │ │ └── .env # Environment variables and secrets |
| 55 | +│ │ ├── staging-main/ |
| 56 | +│ │ │ ├── config.yaml |
| 57 | +│ │ │ ├── provider-profile.yaml |
| 58 | +│ │ │ └── .env |
| 59 | +│ │ └── prod-primary/ |
| 60 | +│ │ ├── config.yaml |
| 61 | +│ │ ├── provider-profile.yaml |
| 62 | +│ │ └── .env |
| 63 | +│ └── outputs/ # GENERATED DATA |
| 64 | +│ ├── environments/ # Generated per environment |
| 65 | +│ │ ├── dev-alice/ |
| 66 | +│ │ │ ├── provision/ # Infrastructure provisioning |
| 67 | +│ │ │ │ ├── terraform/ # Terraform/OpenTofu files |
| 68 | +│ │ │ │ └── cloud-init/ # Cloud-init configurations |
| 69 | +│ │ │ ├── deployment/ # Application deployment |
| 70 | +│ │ │ │ ├── application/ # Application compose and config |
| 71 | +│ │ │ │ │ ├── compose.yaml # Docker Compose file |
| 72 | +│ │ │ │ │ ├── .env # Application environment |
| 73 | +│ │ │ │ │ ├── tracker/ # Tracker service config |
| 74 | +│ │ │ │ │ │ └── tracker.toml |
| 75 | +│ │ │ │ │ ├── nginx/ # Nginx service config |
| 76 | +│ │ │ │ │ │ └── nginx.conf |
| 77 | +│ │ │ │ │ ├── prometheus/ # Prometheus service config |
| 78 | +│ │ │ │ │ │ └── prometheus.yml |
| 79 | +│ │ │ │ │ ├── grafana/ # Grafana service config |
| 80 | +│ │ │ │ │ │ └── grafana.ini |
| 81 | +│ │ │ │ │ ├── mysql/ # MySQL service config |
| 82 | +│ │ │ │ │ │ └── my.cnf |
| 83 | +│ │ │ │ │ └── backups/ # Backup configurations |
| 84 | +│ │ │ │ │ ├── backup-schedule.yaml |
| 85 | +│ │ │ │ │ ├── retention-policy.yaml |
| 86 | +│ │ │ │ │ └── backup-scripts/ |
| 87 | +│ │ │ │ └── scripts/ # Deployment scripts |
| 88 | +│ │ │ └── logs/ # Deployment logs |
| 89 | +│ │ ├── staging-main/ |
| 90 | +│ │ │ ├── provision/ |
| 91 | +│ │ │ │ ├── terraform/ |
| 92 | +│ │ │ │ └── cloud-init/ |
| 93 | +│ │ │ ├── deployment/ |
| 94 | +│ │ │ │ ├── application/ # Application compose and config |
| 95 | +│ │ │ │ │ ├── compose.yaml |
| 96 | +│ │ │ │ │ ├── .env |
| 97 | +│ │ │ │ │ ├── tracker/ |
| 98 | +│ │ │ │ │ ├── nginx/ |
| 99 | +│ │ │ │ │ ├── prometheus/ |
| 100 | +│ │ │ │ │ ├── grafana/ |
| 101 | +│ │ │ │ │ ├── mysql/ |
| 102 | +│ │ │ │ │ └── backups/ |
| 103 | +│ │ │ │ └── scripts/ |
| 104 | +│ │ │ └── logs/ |
| 105 | +│ │ └── prod-primary/ |
| 106 | +│ │ ├── provision/ |
| 107 | +│ │ │ ├── terraform/ |
| 108 | +│ │ │ └── cloud-init/ |
| 109 | +│ │ ├── deployment/ |
| 110 | +│ │ └── logs/ |
| 111 | +│ └── cache/ # Build cache and temporary files |
| 112 | +│ ├── downloads/ # Downloaded assets |
| 113 | +│ └── compiled/ # Compiled templates |
| 114 | +└── src/ # SOURCE CODE (version controlled) |
| 115 | + ├── templates/ # Configuration templates |
| 116 | + │ ├── infrastructure/ # Infrastructure templates |
| 117 | + │ │ ├── terraform/ |
| 118 | + │ │ ├── cloud-init/ |
| 119 | + │ │ └── compose/ |
| 120 | + │ ├── application/ # Application configuration templates |
| 121 | + │ │ ├── tracker/ |
| 122 | + │ │ ├── nginx/ |
| 123 | + │ │ ├── prometheus/ |
| 124 | + │ │ └── grafana/ |
| 125 | + │ └── schemas/ # JSON Schema validation |
| 126 | + │ ├── environment.schema.json |
| 127 | + │ ├── provider.schema.json |
| 128 | + │ └── composite.schema.json |
| 129 | + ├── defaults/ # Base configuration defaults |
| 130 | + │ ├── common.yaml # Universal defaults |
| 131 | + │ ├── development.yaml # Development environment defaults |
| 132 | + │ ├── staging.yaml # Staging environment defaults |
| 133 | + │ └── production.yaml # Production environment defaults |
| 134 | + ├── provider_profiles/ # Provider profile definitions |
| 135 | + │ ├── libvirt.yaml # Local development provider |
| 136 | + │ ├── hetzner.yaml # Hetzner Cloud provider |
| 137 | + │ ├── aws.yaml # AWS provider |
| 138 | + │ └── digitalocean.yaml # DigitalOcean provider |
| 139 | + ├── docs/ # Source code documentation |
| 140 | + │ ├── README.md |
| 141 | + │ ├── configuration.md |
| 142 | + │ ├── deployment.md |
| 143 | + │ └── troubleshooting.md |
| 144 | + └── tests/ # Test suite |
| 145 | + ├── unit/ |
| 146 | + ├── integration/ |
| 147 | + └── fixtures/ |
| 148 | +``` |
| 149 | + |
| 150 | +## 🔧 Configuration Architecture |
| 151 | + |
| 152 | +### Environment Configuration Split |
| 153 | + |
| 154 | +Each environment directory in `inputs/environments/` contains exactly three files: |
| 155 | + |
| 156 | +#### 1. Environment Configuration (`config.yaml`) |
| 157 | + |
| 158 | +Contains environment-specific settings **without secrets**: |
| 159 | + |
| 160 | +```yaml |
| 161 | +# inputs/environments/staging-main/config.yaml |
| 162 | +metadata: |
| 163 | + name: staging-main |
| 164 | + environment_type: staging |
| 165 | + description: "Main staging environment for testing" |
| 166 | + |
| 167 | +general: |
| 168 | + domain: tracker.staging-torrust-demo.com |
| 169 | + floating_ipv4: "78.47.140.132" |
| 170 | + floating_ipv6: "fd00::1" |
| 171 | + ssl: |
| 172 | + enabled: true |
| 173 | + |
| 174 | + database: |
| 175 | + enable_backups: true |
| 176 | + retention_days: 7 |
| 177 | + |
| 178 | +provider_profile: hetzner # References provider_profiles/hetzner.yaml |
| 179 | + |
| 180 | +application: |
| 181 | + tracker: |
| 182 | + enable_stats: true |
| 183 | + log_level: info |
| 184 | + mysql: |
| 185 | + root_password: ${MYSQL_ROOT_PASSWORD} |
| 186 | + tracker_admin_token: ${TRACKER_ADMIN_TOKEN} |
| 187 | + grafana_admin_password: ${GF_SECURITY_ADMIN_PASSWORD} |
| 188 | +``` |
| 189 | +
|
| 190 | +#### 2. Provider Configuration (`provider-profile.yaml`) |
| 191 | + |
| 192 | +Contains provider-specific credentials and settings: |
| 193 | + |
| 194 | +```yaml |
| 195 | +# inputs/environments/staging-main/provider-profile.yaml |
| 196 | +provider_profile: hetzner |
| 197 | +
|
| 198 | +credentials: |
| 199 | + api_token: ${HETZNER_API_TOKEN} |
| 200 | + dns_api_token: ${HETZNER_DNS_API_TOKEN} |
| 201 | +
|
| 202 | +ssh: |
| 203 | + public_key_path: ~/.ssh/staging_ed25519.pub |
| 204 | + private_key_path: ~/.ssh/staging_ed25519 |
| 205 | +
|
| 206 | +server: |
| 207 | + vm_size: cx22 |
| 208 | + location: nbg1 |
| 209 | +``` |
| 210 | + |
| 211 | +#### 3. Environment Variables (`.env`) |
| 212 | + |
| 213 | +```bash |
| 214 | +# inputs/environments/staging-main/.env |
| 215 | +# Hetzner API Credentials |
| 216 | +HETZNER_API_TOKEN=your_staging_api_token_here |
| 217 | +HETZNER_DNS_API_TOKEN=your_staging_dns_token_here |
| 218 | +
|
| 219 | +# Application Secrets |
| 220 | +MYSQL_ROOT_PASSWORD=secure_staging_root_password |
| 221 | +TRACKER_ADMIN_TOKEN=secure_staging_admin_token |
| 222 | +GF_SECURITY_ADMIN_PASSWORD=secure_staging_grafana_password |
| 223 | +``` |
| 224 | + |
| 225 | +## 🚀 Deployment Structure Details |
| 226 | + |
| 227 | +### Application Directory Components |
| 228 | + |
| 229 | +The `deployment/application/` directory contains all the files needed for application deployment: |
| 230 | + |
| 231 | +#### Docker Compose Configuration |
| 232 | + |
| 233 | +- **`compose.yaml`**: Main Docker Compose file defining all services |
| 234 | +- **`.env`**: Environment variables for Docker Compose services |
| 235 | + |
| 236 | +#### Service-Specific Configuration |
| 237 | + |
| 238 | +- **`tracker/`**: Torrust Tracker configuration files |
| 239 | + |
| 240 | + - `tracker.toml`: Main tracker configuration |
| 241 | + - Custom settings for announce intervals, API tokens, etc. |
| 242 | + |
| 243 | +- **`nginx/`**: Reverse proxy configuration |
| 244 | + |
| 245 | + - `nginx.conf`: Main nginx configuration |
| 246 | + - SSL certificate management, routing rules |
| 247 | + |
| 248 | +- **`prometheus/`**: Metrics collection configuration |
| 249 | + |
| 250 | + - `prometheus.yml`: Scraping targets and rules |
| 251 | + - Alert rules and retention policies |
| 252 | + |
| 253 | +#### Backup Configuration |
| 254 | + |
| 255 | +- **`backups/`**: Centralized backup management |
| 256 | + - `backup-schedule.yaml`: Automated backup schedules |
| 257 | + - `retention-policy.yaml`: Data retention rules |
| 258 | + - `backup-scripts/`: Custom backup automation scripts |
| 259 | + |
| 260 | +### Scripts Directory |
| 261 | + |
| 262 | +- **`scripts/`**: Deployment and maintenance scripts |
| 263 | + - Environment-specific deployment automation |
| 264 | + - Health check and monitoring scripts |
| 265 | + - Rollback and recovery procedures |
| 266 | + |
| 267 | +## 🚀 Workflow Examples |
| 268 | + |
| 269 | +### Initial Setup |
| 270 | + |
| 271 | +```bash |
| 272 | +# 1. Clone main application repository |
| 273 | +git clone https://github.com/torrust/torrust-tracker-automation |
| 274 | +cd torrust-tracker-automation |
| 275 | +
|
| 276 | +# 2. Initialize user directories |
| 277 | +mkdir -p data/inputs/environments |
| 278 | +mkdir -p data/outputs/environments data/outputs/cache |
| 279 | +``` |
| 280 | + |
| 281 | +### Environment Deployment |
| 282 | + |
| 283 | +```bash |
| 284 | +# Create new environment |
| 285 | +./src/tools/init-environment.sh staging-main hetzner |
| 286 | +
|
| 287 | +# This creates: |
| 288 | +# - data/inputs/environments/staging-main/config.yaml (template) |
| 289 | +# - data/inputs/environments/staging-main/provider-profile.yaml (template) |
| 290 | +# - data/inputs/environments/staging-main/.env (template) |
| 291 | +``` |
| 292 | + |
| 293 | +### Configuration and Deployment |
| 294 | + |
| 295 | +```bash |
| 296 | +# 1. Edit configuration files |
| 297 | +vim data/inputs/environments/staging-main/config.yaml |
| 298 | +vim data/inputs/environments/staging-main/provider-profile.yaml |
| 299 | +vim data/inputs/environments/staging-main/.env |
| 300 | +
|
| 301 | +# 2. Validate configuration |
| 302 | +./src/tools/validate.sh staging-main |
| 303 | +
|
| 304 | +# 3. Generate deployment files |
| 305 | +./src/tools/configure.sh staging-main |
| 306 | +
|
| 307 | +# 4. Deploy infrastructure and application |
| 308 | +./src/tools/deploy.sh staging-main |
| 309 | +``` |
| 310 | + |
| 311 | +## 🔍 Benefits of This Structure |
| 312 | + |
| 313 | +### 1. Clear Separation of Concerns |
| 314 | + |
| 315 | +- **Source Code**: Lives in `src/`, version controlled with main repository |
| 316 | +- **User Inputs**: Lives in `inputs/`, user-managed configuration data |
| 317 | +- **Generated Outputs**: Lives in `outputs/`, temporary and regenerable |
| 318 | + |
| 319 | +### 2. Security |
| 320 | + |
| 321 | +- Secrets never committed to main repository |
| 322 | +- Provider credentials isolated in `provider-profile.yaml` files |
| 323 | +- Environment variables kept separate from configuration |
| 324 | +- Clear boundaries between public and private data |
| 325 | + |
| 326 | +### 3. Multi-Environment Support |
| 327 | + |
| 328 | +- Each environment is completely isolated |
| 329 | +- One-to-one mapping between environment and provider profile |
| 330 | +- No credential conflicts between environments |
| 331 | +- Easy to add/remove environments |
| 332 | + |
| 333 | +### 4. Operational Excellence |
| 334 | + |
| 335 | +- Comprehensive backup and recovery procedures |
| 336 | +- Standardized deployment workflows |
| 337 | +- Environment-specific customization capabilities |
| 338 | +- Audit trail for configuration changes |
| 339 | + |
| 340 | +## 📝 Main Git Repository .gitignore |
| 341 | + |
| 342 | +The main repository includes a `.gitignore` that excludes the entire data directory: |
| 343 | + |
| 344 | +```gitignore |
| 345 | +# User data and generated outputs (not included in main repo) |
| 346 | +data/ |
| 347 | +
|
| 348 | +# Existing excludes |
| 349 | +*.log |
| 350 | +*.tmp |
| 351 | +.env |
| 352 | +.terraform/ |
| 353 | +terraform.tfstate* |
| 354 | +.DS_Store |
| 355 | +node_modules/ |
| 356 | +``` |
| 357 | + |
| 358 | +### Why Exclude data/ from Main Repository? |
| 359 | + |
| 360 | +1. **Security**: Prevents accidental commit of secrets and credentials to public repo |
| 361 | +2. **Flexibility**: Users can choose their own version control strategy for configurations |
| 362 | +3. **Separation**: Keeps application code separate from user configuration |
| 363 | +4. **Privacy**: User environments and secrets don't need to be public |
| 364 | + |
| 365 | +## 📋 Conclusion |
| 366 | + |
| 367 | +This structure provides a clean foundation for scalable, secure, and maintainable Torrust |
| 368 | +Tracker deployments while supporting diverse user needs and deployment scenarios. |
| 369 | + |
| 370 | +The separation of source code from user data enables both individual developers and |
| 371 | +enterprise teams to use the same automation tooling while maintaining their preferred |
| 372 | +approaches to configuration management and security. |
0 commit comments