Cleanup: remove the dead ML system, fix three broken services, halve the disk use

.github/workflows/tests.yml

@@ -41,16 +41,13 @@ jobs:
         python -m pip install --upgrade pip
         pip install -r requirements-dev.txt
 
-    - name: Lint with flake8
+    - name: Lint with ruff
       run: |
-        # Stop the build if there are Python syntax errors or undefined names
-        flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
-        # Exit-zero treats all errors as warnings
-        flake8 src/ --count --exit-zero --max-complexity=10 --max-line-length=100 --statistics
+        ruff check src/ scripts/ tests/
 
     - name: Check code formatting with black
       run: |
-        black --check src/ tests/
+        black --check src/ scripts/ tests/
 
     - name: Run tests with pytest
       run: |
@@ -66,7 +63,9 @@ jobs:
         fail_ci_if_error: false
       continue-on-error: true
 
-  lint:
+  # Advisory only: the codebase is not fully annotated, so mypy findings are
+  # informational. Ruff is the gating linter and runs in the test job.
+  typecheck:
     runs-on: ubuntu-latest
 
     steps:
@@ -83,11 +82,6 @@ jobs:
         python -m pip install --upgrade pip
         pip install -r requirements-dev.txt
 
-    - name: Lint with pylint
-      run: |
-        pylint src/ --exit-zero --max-line-length=100
-      continue-on-error: true
-
     - name: Type check with mypy
       run: |
         mypy src/ --ignore-missing-imports --no-strict-optional
@@ -112,10 +106,14 @@ jobs:
 
     - name: Check for syntax errors
       run: |
-        python -m py_compile src/*.py
-        python -m py_compile tests/*.py
+        python -m py_compile src/*.py scripts/*.py tests/*.py
 
-    - name: Verify imports (without camera hardware)
+    - name: Verify modules import the way systemd runs them
       run: |
-        python -c "import sys; sys.path.insert(0, 'src'); from logging_config import get_logger; print('✓ logging_config imports successfully')"
-        echo "✓ All non-hardware modules can be imported"
+        # The units run `python3 src/auto_timelapse.py`, so sys.path[0] is src/
+        # and the flat half of the dual-import idiom is what production uses.
+        # Importing as `src.x` in CI would never exercise that path.
+        cd src
+        for module in logging_config config_utils colors exposure weather database; do
+          python -c "import $module" && echo "ok $module"
+        done

.gitignore

@@ -16,8 +16,16 @@ build/
 coverage.xml
 htmlcov/
 .mypy_cache/
+.ruff_cache/
 .tox/
 
+# Agent/editor session state
+.claude/
+
+# Rendered by scripts/install.sh before it was fixed to use a temp dir.
+# Kept so an old stray copy can never be committed by accident.
+/raspilapse.service
+
 # Output directories (runtime generated)
 test_photos/
 test_videos/

.pre-commit-config.yaml

@@ -2,20 +2,24 @@
 # Install: pip install pre-commit
 # Setup: pre-commit install
 # Run manually: pre-commit run --all-files
+#
+# The black rev here must match requirements-dev.txt and pyproject.toml.
+# Black's stable style changes between yearly releases, so a mismatch means
+# locally-formatted code gets rejected by CI. Line length comes from
+# pyproject.toml, not from args here, so there is one place to change it.
 
 repos:
   - repo: https://github.com/psf/black
-    rev: 24.10.0
+    rev: 25.11.0
     hooks:
       - id: black
         language_version: python3
-        args: ['--line-length=100']
 
-  - repo: https://github.com/PyCQA/flake8
-    rev: 6.1.0
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.13
     hooks:
-      - id: flake8
-        args: ['--max-line-length=100', '--extend-ignore=E203,W503']
+      - id: ruff
+        args: [--fix]
 
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0

CHANGELOG.md

@@ -5,6 +5,117 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.0] - 2026-07-26
+
+A cleanup pass over the whole project. Behaviour is unchanged apart from
+highlight protection, which is new and can be turned off in config.
+
+### Removed
+- **The ML exposure system.** It had not run since January: `_init_ml_predictor()`
+  returned early whenever `direct_brightness_control` was true. With ML inert the
+  "legacy" branches of `get_camera_settings` were unreachable too, and the formula
+  functions behind them had one live caller left -- the metadata diagnostics, which
+  re-ran the entire exposure calculation to fill in a JSON field.
+  Gone: `ml_exposure.py`, `ml_exposure_v2.py`, `bootstrap_ml.py`,
+  `bootstrap_ml_v2.py`, `ML.md`, `ml_state/` and four test modules.
+- **`analyze_timelapse.py`** (1,967 lines). It read per-frame metadata JSON that
+  cleanup deletes after 7 days, so it could never look back further than a week,
+  while `scripts/db_graphs.py` reads months from the database. Its one
+  non-overlapping chart, white balance, is now `create_white_balance_graph()`.
+- **`manuals/*.pdf`** -- 45 MB of third-party PDFs nothing referenced.
+- **A leaked Codecov token** from `docs/MAINTAINER.md`. Rotate it if you forked
+  this repo before now; it remains in history.
+- **`ml_state/ml_state.json`**, which shipped one camera's learned model to
+  everyone who cloned.
+- Five installer scripts, `test.sh`, `check_disk_space.sh`,
+  `check_capture_rate.sh`, and twelve documentation files.
+
+### Added
+- **Highlight protection** (`adaptive_timelapse.highlight_protection`). Lowers the
+  brightness target when the top of the histogram nears clipping, so bright skies
+  keep detail. Off at night by default. `enabled: false` reverts.
+  Not to be confused with the p95 protection listed under 1.3.0: that one scaled
+  the *exposure*, lived in the now-deleted ML path, and never reached the camera.
+  This one scales the *target*, which is what makes its equilibrium independent
+  of `brightness_damping`.
+- **`scripts/install.sh` as the single entry point**, with `--only`, `--check`,
+  `--dry-run`, `--uninstall` and `--with-watchdog`. It renders `systemd/*.in`
+  templates rather than copying units that hardcode `pi` and `/home/pi`.
+- **`database.retention_days`** and `python3 src/database.py --prune|--vacuum|--stats`.
+  Defaults to 0, keep everything; the example ships 180 days.
+- **`src/exposure.py`** and **`src/config_utils.py`**.
+- **`tests/test_config_example.py`**, which fails when the example config and the
+  code that reads it drift apart in either direction.
+
+### Fixed
+- **The daily-video service could not run.** `upload_service.py` imported
+  `requests_toolbelt` unguarded and `/usr/bin/python3` did not have it. Guarded,
+  with a `requests.post` fallback.
+- **An empty day failed the unit.** `make_timelapse.py` now exits 2 for "no images"
+  and `daily_timelapse.py` maps that to success.
+- **The upload retry queue retried the impossible.** 172 rows, all pending since
+  January, all pointing at videos deleted long ago, and no installer had ever
+  installed the timer that drains them. Rows whose source is gone are cancelled;
+  `failed` is treated as terminal; `--purge-missing` clears a backlog.
+- **The installed daily-video timer had drifted** from the repo: `Requires=`, two
+  `OnCalendar=` lines and `Persistent=true`, all removed months ago and never
+  redeployed, which is why it fired at boot and failed.
+- **Every log line was stored twice**, once in `logs/` and once in the journal.
+  `logging.console` becomes tri-state; `auto` skips the console handler under
+  systemd. journald is capped at 200 MB.
+- **Logging ignored `-c/--config`.** Seven modules call `get_logger()` at import
+  time, before argparse runs; `configure_logging()` now reconfigures them.
+- **Weather hammered its endpoint.** The 300 s cache was per-instance and the
+  instance was rebuilt twice per capture cycle, so it never applied. There was no
+  backoff either: one outage produced 72,536 identical error lines. Also fixed
+  `data.get("data", {})` returning `None` on `"data": null`, which was 2,204 more.
+- **`brightness_p25` and `brightness_p75` were NULL on every row** ever written --
+  the producer emitted p10/p90.
+- **The upload queue schema was defined twice** and had drifted, leaving the live
+  database pinned at v3 with the v4 index missing.
+- **18 tests had never run**, in four classes shadowed by a later class of the
+  same name.
+
+### Changed
+- SQLite runs in WAL mode. Three unused indexes dropped (26 MB on a 515k-row
+  database, three fewer B-tree writes per capture).
+- `auto_timelapse.py` is under 1,000 lines, down from 3,230.
+- ruff replaces flake8 and pylint, and the CI lint step can now fail -- it was
+  `--exit-zero` *and* `continue-on-error`.
+- black pinned to one version across `requirements-dev.txt`, `pyproject.toml` and
+  `.pre-commit-config.yaml`. The mismatch was the cause of the recurring CI
+  formatting failures.
+- `pyproject.toml` version is now read from `src/__version__.py`, and its
+  dependency list matches what the code imports.
+- `graph_ml_patterns.py` -> `scripts/graph_solar_patterns.py`; it was never ML.
+- The `timelapse:` and `graphs:` config blocks are gone -- no code read either,
+  and `timelapse.interval: 3` sat next to `adaptive_timelapse.interval: 30`.
+
+### Migrating
+
+Existing installs:
+
+```bash
+git pull
+./scripts/install.sh --check
+./scripts/install.sh              # redeploys the corrected units
+sudo systemctl restart raspilapse
+```
+
+Then, in `config/config.yml`:
+
+- set `logging.console: auto`
+- optionally add `adaptive_timelapse.highlight_protection` (see the example)
+- optionally set `database.retention_days` -- absent means keep everything
+- remove `adaptive_timelapse.ml_exposure` and `direct_brightness_control`,
+  both now inert
+
+If uploads were configured, clear any stale queue:
+
+```bash
+python3 src/retry_uploads.py --purge-missing
+```
+
 ## [1.3.2] - 2026-01-23
 
 ### Fixed
@@ -550,7 +661,7 @@ Raspilapse v1.0.0 is production-ready for year-long operation.
 3. Documentation moved to `docs/` folder
 4. No configuration changes required
 
-See [docs/V1_RELEASE_NOTES.md](docs/V1_RELEASE_NOTES.md) for complete release details.
+See the v1.0.0 release notes on GitHub for complete release details.
 
 ---
 

CITATION.cff

@@ -2,8 +2,8 @@ cff-version: 1.2.0
 message: "If you use this software, please cite it as below."
 type: software
 title: "Raspilapse: Raspberry Pi Camera Timelapse Library"
-version: 0.9.0-beta
-date-released: 2025-11-05
+version: 1.4.0
+date-released: '2026-07-26'
 url: "https://github.com/ekstremedia/raspilapse"
 repository-code: "https://github.com/ekstremedia/raspilapse"
 license: MIT

CONTRIBUTING.md

@@ -0,0 +1,84 @@
+# Contributing to Raspilapse
+
+Thanks for your interest in the project. Bug reports, config examples from
+other latitudes, and pull requests are all welcome.
+
+## Development setup
+
+```bash
+git clone https://github.com/ekstremedia/raspilapse.git
+cd raspilapse
+pip3 install -r requirements-dev.txt
+
+# Optional but recommended: auto-formats on commit
+pip3 install pre-commit && pre-commit install
+```
+
+You do not need a Raspberry Pi to run the test suite. `picamera2` is imported
+lazily and stubbed in tests; only one test is hardware-gated.
+
+## Before you commit
+
+```bash
+make format    # black
+make lint      # ruff
+make check     # black --check
+make test      # pytest
+```
+
+or `make all`, which runs all four.
+
+**Use the black version pinned in `requirements-dev.txt`.** Black's stable
+style changes between yearly releases, so a newer black on your machine will
+reformat files that CI then rejects. Installing the dev requirements gives you
+the right one; the pre-commit hook uses whatever `black` is on your `PATH`, so
+check it with `black --version` if CI disagrees with you.
+
+## Pull requests
+
+1. Branch from `main` (`git checkout -b feature/thing`).
+2. Make the change, with tests.
+3. Run `make all`.
+4. Open the PR. Explain what problem it solves, not just what it changes.
+
+Code standards: docstrings on public functions, tests for new behaviour,
+line length 100 (enforced by black and ruff via `pyproject.toml`).
+
+## Project layout
+
+| Path | Contents |
+|------|----------|
+| `src/` | Application code. Every module is importable both as `src.x` and as bare `x` — the systemd units run scripts directly, so `sys.path[0]` is `src/`. |
+| `scripts/` | Installer and operator tools (shell + standalone Python) |
+| `systemd/` | Unit templates (`*.in`, substituted by `scripts/install.sh`) |
+| `config/` | `config.example.yml` is the documented schema; `config.yml` is gitignored |
+| `tests/` | pytest suite, one module per `src/` module (`__version__.py` is covered by `test_version.py`) |
+| `docs/` | User documentation |
+
+Never commit `config/config.yml` — it holds API keys. `.gitignore` covers it,
+but check `git status` before you push.
+
+## Releasing
+
+Maintainers only.
+
+1. `src/__version__.py` is the single source of truth for the version.
+   `pyproject.toml` reads it dynamically; update `CITATION.cff` by hand.
+2. Add a `CHANGELOG.md` entry under a new `## [x.y.z]` heading.
+   `tests/test_version.py` asserts these two agree.
+3. `make all` must pass.
+4. Commit, then tag: `git tag vX.Y.Z && git push --tags`.
+5. Create the GitHub release from the tag, pasting the CHANGELOG section.
+
+Semantic versioning: major for breaking changes, minor for features, patch
+for fixes.
+
+### CI secrets
+
+`CODECOV_TOKEN` lives in **Settings → Secrets and variables → Actions**.
+Get its value from the Codecov repository settings page — never paste a token
+into a file in this repository, including documentation.
+
+---
+
+Thank you for contributing.