GeneWeb Legacy Restoration Project

🎯 Mission

Restore, secure, test, and deploy the legacy OCaml genealogy software GeneWeb (1995-2008) without altering its core behavior.

🚀 Quick Start

What is GeneWeb?

GeneWeb is an open-source genealogy application written in OCaml that:

• Imports/exports GEDCOM files (genealogical data format)
• Stores data in proprietary .gwb binary databases
• Serves family tree web pages via the gwd daemon
• Provides admin UI via gwsetup daemon

Prerequisites

• macOS or Linux (x86_64)
• Python 3.11+ (for running tests and Python proxy server)
• Basic terminal/command-line knowledge
• A web browser

📦 Getting Started

There are two ways to use GeneWeb depending on your needs:

---

Option A: Quick Start for Testing (Recommended)

This is what CI uses and what you need for running Python tests:

# 1. Clone the repository
git clone https://github.com/Antonyjin/Legacy-Project.git
cd Legacy-Project

# 2. Start gwd on test port (23179)
cd GeneWeb
./gw/gwd -hd ./gw -bd ./bases -p 23179 -lang en &
cd ..

# 3. Verify it's running
curl http://localhost:23179/test
# Should return HTML with "GeneWeb"

# 4. Setup Python virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate  # On macOS/Linux
pip install -r requirements.txt

# 5. Run Python tests
pytest tests/python/ -v

# 6. Stop gwd when done
pkill -f gwd

# 7. Deactivate venv when done (optional)
deactivate

Access test database:

• 🌐 Main app: http://localhost:23179/test
• 👤 Person page: http://localhost:23179/test?p=Charles&n=Windsor
• 📊 Statistics: http://localhost:23179/test?m=STAT

Database: test.gwb contains British Royal Family data (188 persons)

---

Option B: Manual Launch for Exploration

If you want to explore GeneWeb manually (not for testing):

macOS/Linux - Manual Start (Recommended):

cd Legacy-Project/GeneWeb

# Start gwd (web server) on port 2317
./gw/gwd -hd ./gw -bd ./bases -p 2317 -lang en &

# Start gwsetup (admin interface) on port 2316 - optional
./gw/gwsetup -gd ./gw -lang en &

# Access application
echo "Main app: http://localhost:2317/test"
echo "Admin: http://localhost:2316"

macOS - Using geneweb.sh (Alternative):

cd GeneWeb
./geneweb.sh
# Note: This legacy script may have path issues. Use manual start above if it fails.

Access application:

• 🌐 Main app: http://localhost:2317/test
• ⚙️ Admin panel: http://localhost:2316
• 📄 Landing page: Open GeneWeb/START.htm in browser

Stop servers:

pkill -f gwd
pkill -f gwsetup

---

Exploring the Database

Try these pages (adjust port as needed: 23179 for testing, 2317 for manual):

• 🏠 Home: http://localhost:23179/test
• 👤 Person: http://localhost:23179/test?p=Charles&n=Windsor
• 👨‍👩‍👧‍👦 Family: http://localhost:23179/test?m=F&p=charles&n=windsor
• 📅 Calendar: http://localhost:23179/test?m=CAL
• 📝 First names: http://localhost:23179/test?m=P
• 📋 Surnames: http://localhost:23179/test?m=N
• 📊 Statistics: http://localhost:23179/test?m=STAT

🧪 Running Tests

Prerequisites for Testing

Create and activate a virtual environment (recommended):

# Create virtual environment
python3 -m venv venv

# Activate it
source venv/bin/activate  # On macOS/Linux
# On Windows: venv\Scripts\activate

# Install Python dependencies
pip install -r requirements.txt

# Verify installation
pip list | grep -E "(pytest|flask|requests)"

Or use the setup script:

./python_app/setup_venv.sh
source venv/bin/activate

Note: Always activate the virtual environment before running tests or the Python server.

Python Tests (Unit + Integration + Functional)

We have comprehensive Python tests that validate OCaml behavior:

# From project root (Legacy-Project/)

# Run all Python tests
pytest tests/python/ -v

# Run with coverage
pytest tests/python/ --cov=tests/python --cov-report=html

# Run specific test type
pytest tests/python/unit/ -v          # Unit tests- ✅ CI BLOCKING
pytest tests/python/integration/ -v   # Integration tests ✅ CI BLOCKING
pytest tests/python/functional/ -v    # Functional tests

# Run by marker
pytest -m unit        # Unit tests only
pytest -m integration # Integration tests only
pytest -m functional  # Functional tests only

Test Status:

• ✅ Unit Tests: Test OCaml behavior via HTTP/Python utilities — CI blocks on failure
• ⚠️ Integration Tests: gwd lifecycle, port conflicts, concurrency — CI blocks
• ⚠️ Functional Tests: End-to-end user workflows — non‑blocking during development

⚠️ Integration Test Issues (Process Management)

Problem: Some integration tests fail due to OCaml gwd daemonization behavior.

Root Cause: OCaml gwd daemonizes (parent process exits immediately after forking), causing tests to fail with:

RuntimeError: gwd exited unexpectedly while starting

Solution: Apply the process management fix from IT-PY-003 to affected test files:

• Replace proc.poll() checks with HTTP readiness checks
• Use pkill for reliable cleanup
• Check is_running() instead of process status

Files Needing Fix:

• test_html_generation.py (10 tests)
• test_authentication.py (9 tests)
• test_error_handling.py (13 tests)
• test_localization.py (9 tests)
• test_performance.py (11 tests)

Golden Tests (Optional)

Golden tests validate that OCaml output remains unchanged. They are optional and disabled by default in CI; run them locally or on-demand in CI.

# From project root
export LC_ALL=C.UTF-8 TZ=UTC
chmod +x ./scripts/golden/run_golden.sh
./scripts/golden/run_golden.sh validate

What this tests:

• GEDCOM export consistency
• HTML rendering stability (10 page types)
• Data normalization (whitespace, timestamps, random IDs)

Note: Golden tests are disabled by default in CI and can be triggered on demand (see .github/workflows/ci.yml).

📚 Documentation

For Quick Testing & Understanding

1. This README - Quick start
2. Wiki Home - Full documentation
3. Product Runbook - How to run GeneWeb
4. OCaml Overview - Understanding the codebase
5. CI Workflow Guide - Understanding CI setup
6. Code Quality Guide - Running quality checks locally
7. Branch Protection Setup - Protect main with CI checks
8. Golden Tests - Create/validate golden references

For Deep Dives

• Test Policy - QA strategy
• Test Protocols - Test types (UT/IT/FT)
• Architecture - System design

Test-Specific Documentation

• Unit Test README - Python unit testing strategy
• Integration Test README - Integration testing guide
• Python Proxy Server - Proxy server documentation

🔍 Understanding the Codebase

Directory Structure

Legacy-Project/
├── GeneWeb/                    # Main OCaml application (prebuilt binaries)
│   ├── gw/                     # Binaries (gwd, gwsetup, ged2gwb, gwb2ged)
│   ├── bases/                  # Databases (test.gwb with 188 persons)
│   ├── geneweb.sh             # Launcher script (macOS)
│   └── START.htm              # Landing page
├── scripts/                    # Test and utility scripts
│   └── golden/                # Golden test harness (currently disabled)
│       ├── run_golden.sh      # Main golden test script
│       └── test_gedcom_import.sh  # GEDCOM roundtrip test
├── tests/                      # Python tests
│   ├── python/
│   │   ├── unit/              # Unit tests ✅
│   │   ├── integration/       # Integration tests ⚠️
│   │   └── functional/        # Functional tests ⚠️
│   └── golden/                # Golden references (disabled)
│       └── goldens/v1/        # Versioned golden snapshots
├── docs/                       # Project documentation
│   └── CI_WORKFLOW_GUIDE.md   # CI setup guide
├── .github/workflows/         # CI/CD pipelines
│   └── ci.yml                 # Main CI workflow
├── pytest.ini                 # Pytest configuration
├── .coveragerc                # Coverage configuration
└── requirements.txt           # Python dependencies

Key OCaml Binaries

Binary	Purpose	Example Usage
gwd	Web daemon (serves pages)	gwd -hd ./gw -bd ./bases -p 2317 -lang en
gwsetup	Admin interface	gwsetup -gd ./gw -lang en
ged2gwb	GEDCOM → GeneWeb import	ged2gwb input.ged -o bases/mybase
gwb2ged	GeneWeb → GEDCOM export	gwb2ged bases/mybase.gwb -o output.ged
gwu	Text export utility	gwu bases/mybase.gwb > dump.gw

🧪 Test Types

We have 3 test categories (4th coming soon):

1. ✅ Unit Tests - Test OCaml functions via HTTP/Python — Blocks CI
2. ⚠️ Integration Tests - gwd lifecycle, ports, concurrency — Blocks CI
3. ⚠️ Functional Tests - End-to-end user workflows — In development (non‑blocking)
4. 🔄 Golden Tests - Regression detection — Optional/on‑demand

See Test Protocols and CI Workflow Guide for details.

🐛 Troubleshooting

Port already in use

# macOS/Linux - Find and kill processes
lsof -ti:2317 | xargs kill -9
lsof -ti:2316 | xargs kill -9

# Or use pkill
pkill -f gwd
pkill -f gwsetup

Python tests failing

# Make sure gwd is running on the correct port
ps aux | grep gwd

# Check if test database exists
ls -la GeneWeb/bases/test.gwb/

# Verify Python dependencies
pip list | grep pytest

"cannot execute binary file" on Linux

• The bundled binaries in GeneWeb/ are for macOS only
• Download Linux binaries from GeneWeb releases
• Follow the Linux setup instructions above

"Failed - unbound var sosa_ref.key"

• Tree and search pages require Sosa reference configuration
• Use gwsetup to configure a Sosa root person
• Access gwsetup at http://localhost:2316
• See ISSUE_50_SKIPPED.md for details

Golden tests failing with whitespace differences

# Regenerate goldens if changes are intentional
export LC_ALL=C.UTF-8 TZ=UTC
./scripts/golden/run_golden.sh create

# Note: Golden tests are currently disabled in CI
# They will be re-enabled during Week 2-3 migration phase

Python import errors

# Make sure you're in the project root
cd Legacy-Project

# Install dependencies
pip install -r requirements.txt

# Verify pytest works
pytest --version

🎓 Learning Path

Day 1: Understand the basics

1. Read this README
2. Launch GeneWeb with ./GeneWeb/geneweb.sh (macOS) or follow Linux instructions
3. Explore the test database pages
4. Read OCaml Overview

Day 2: Understand the tests

1. Install Python dependencies: pip install -r requirements.txt
2. Run Python tests: pytest tests/python/ -v
3. Review Test Protocols
4. Check CI Workflow Guide
5. Check CI results on GitHub Actions

Day 3: Understand the architecture

1. Read Architecture
2. Read Runbook
3. Try importing your own GEDCOM file

📊 Test and CI Notes

• Unit and integration tests are enforced in CI; functional tests are allowed to fail during development.
• Coverage is reported; thresholds may not be enforced yet.
• See docs/CI_WORKFLOW_GUIDE.md for current details.

🐍 Python Proxy Server (MIG-INF-001)

New! Python Flask proxy server with backend toggle (OCaml/Python).

See python_app/README.md for full documentation.

Quick start:

# Setup virtual environment (if not done)
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Run server with OCaml backend (default)
BACKEND=ocaml python -m python_app.app

# Run with Python backend (uses migrated functions)
BACKEND=python python -m python_app.app

Access: http://localhost:2318/test

Issue: #225 - [MIG-INF-001] Create Python proxy server with BACKEND toggle

---

🤝 Contributing

1. Create a feature branch: git checkout -b feature-name
2. Setup virtual environment: python3 -m venv venv && source venv/bin/activate
3. Install dependencies: pip install -r requirements.txt
4. Make changes and test locally: pytest tests/python/ -v
5. Commit with conventional commits: feat:, fix:, docs:, etc.
6. Push and create a PR
7. CI will automatically run all tests

Branch Protection Rules:

• ✅ Unit tests must pass
• ✅ Integration tests must pass
• ⚠️ Functional tests may fail (in development)

📞 Support

• Wiki: https://github.com/Antonyjin/Legacy-Project/wiki
• Issues: https://github.com/Antonyjin/Legacy-Project/issues
• Upstream GeneWeb: https://geneweb.tuxfamily.org/
• CI Status: https://github.com/Antonyjin/Legacy-Project/actions

📜 License

GeneWeb is distributed under the GNU General Public License. See LICENSE.txt.

---

Quick Command Reference

# ==========================================
# Start GeneWeb - FOR TESTING (Port 23179)
# ==========================================

# Start gwd (what CI uses)
cd Legacy-Project/GeneWeb
./gw/gwd -hd ./gw -bd ./bases -p 23179 -lang en &
cd ..

# Verify it's running
curl http://localhost:23179/test

# Stop
pkill -f gwd

# ==========================================
# Start GeneWeb - FOR MANUAL USE (Port 2317)
# ==========================================

# Recommended manual start
cd Legacy-Project/GeneWeb
./gw/gwd -hd ./gw -bd ./bases -p 2317 -lang en &
./gw/gwsetup -gd ./gw -lang en &

# Alternative: Use legacy script (may have issues)
./geneweb.sh

# Stop
pkill -f gwd
pkill -f gwsetup

# ==========================================
# Python Tests (from project root)
# ==========================================

# IMPORTANT: Start gwd on port 23179 first (see above)

# Setup virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate  # On macOS/Linux
pip install -r requirements.txt

# Run all tests
pytest tests/python/ -v

# Deactivate venv when done
# deactivate

# Run by type
pytest tests/python/unit/ -v              # Unit tests
pytest tests/python/integration/ -v       # Integration tests
pytest -m unit                            # Unit marker
pytest -m integration                     # Integration marker

# With coverage
pytest tests/python/ --cov=tests/python --cov-report=html

# Specific test file
pytest tests/python/unit/test_http_params.py -v

# ==========================================
# Golden Tests (disabled, for reference)
# ==========================================

# Validate (when re-enabled)
export LC_ALL=C.UTF-8 TZ=UTC
./scripts/golden/run_golden.sh validate

# Regenerate goldens
./scripts/golden/run_golden.sh create

# ==========================================
# GEDCOM Import/Export
# ==========================================

# Export GEDCOM
GeneWeb/gw/gwb2ged GeneWeb/bases/test.gwb -o export.ged

# Import GEDCOM
GeneWeb/gw/ged2gwb input.ged -o GeneWeb/bases/newbase

# ==========================================
# Check Status
# ==========================================

# Check if gwd is running
ps aux | grep gwd

# Check CI status
# Visit: https://github.com/Antonyjin/Legacy-Project/actions

# Check test database
ls -la GeneWeb/bases/test.gwb/

# Check Python installation
python --version  # Should be 3.11+
pytest --version

# ==========================================
# Python Proxy Server (MIG-INF-001)
# ==========================================

# Setup (one-time)
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Run with OCaml backend (default)
BACKEND=ocaml python -m python_app.app

# Run with Python backend
BACKEND=python python -m python_app.app

# Access: http://localhost:2318/test

Need help? Start with the Wiki Home, check CI Workflow Guide, or ask in Issues! 🚀

---

📚 Additional Documentation

Deployment & Operations

• Deployment Guide - Complete guide for local, Docker, and production deployment
• CI/CD Guide - How the CI/CD pipeline works and how to debug issues

Architecture & Technical Decisions

The following architectural decision records document all major decisions made during the project:

• ADR-004: Migration Strategy - Phased, test-driven approach with black-box testing
• ADR-005: Testing Approach - Four-tier testing strategy (unit, integration, golden, smoke)
• ADR-006: Deployment Platform - Docker on VPS with docker-compose
• ADR-007: CI/CD Pipeline - GitHub Actions-based automation