Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions docs/repo/415-git-weight-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# #415 Repo-Weight Audit — verified state on `18b4d023`

**Author**: po-2024 (worker) · **Date**: 2026-07-01 · **Base**: master `18b4d023`
**Status**: **AUDIT / STALE-ISSUE REFRESH**. Read-only. 0 write.
**Reproducibility**: [`tools/git-weight-audit.py`](../../tools/git-weight-audit.py) — `python tools/git-weight-audit.py`.

## TL;DR — #415 Phase 1 is CLOSED; the issue body is stale on that point

The #415 issue body (measured on `0bf77852`, 2026-06-01) states *"~1,4 GB de binaires encore trackés au HEAD"* and proposes Phase 1 (gitignore + `git rm --cached`). **Phase 1 has since been executed and merged**:

- **PR #416** `94b43712` — *"stop tracking 968 MB of regenerable build artifacts (#415 Phase 1)"*
- **PR #501** `ff031470` — *"#415 untrack 10.5 MB 2sxc install module + gitignore"*

The audit confirms it on current master: the regenerable zones are **0 bytes at HEAD** (untracked) — they survive only in history.

| Regenerable zone | In history | At HEAD |
|---|---:|---:|
| `Published/` .NET builds (osx/linux/win zips, .exe) | **1.2 GB** (42 paths) | **0** ✅ untracked |
| `DNNPlatform/.../Downloads/*.zip` (Print&Play) | 206 MB (12) | **0** ✅ |
| `2sxc/DNN *.resources` install pkgs | 152 MB (197) | **0** ✅ |
| Mindmap SVGs | 48 MB (10) | 48 MB (still tracked — see note) |

→ **Phase 1 is done.** The remaining `.git` weight is **100 % historical** — only a history rewrite (Phase 2) reduces the clone.

## Current measurements (`git count-objects -vH`)

- **`size-pack` = 2.05 GiB** (3 packs, 25 857 objects) — ≈ unchanged from the issue's 2.01 GiB. **Phase 1 does not reduce the pack** (it only stops re-adding at HEAD); this is expected and stated in the issue.
- HEAD checkout ≈ **778 MB** (down from the issue's higher figure once Published/ was untracked).

## What Phase 2 (`git filter-repo`) would reclaim

The top history blobs (deduped by path, largest version) are all in the now-untracked regenerable zones:

- `DNNPlatform/Portals/1/Downloads/Argumentum_Print&Play.zip` — **89 MB**
- `Published/v1.3/osx-x64.zip` — 80–83 MB (multiple versions)
- `Published/v1.3/linux-x64.zip` — 79–81 MB (multiple)
- `Cartes/.../Published/v1/win-x64/Argumentum.AssetConverter.exe` — 79 MB (legacy tree)
- `Published/v1.3/win-x64.zip` + `.001/.002` splits — 50–72 MB

Purging `Published/`, `Cartes/.../Published/`, `Downloads/*.zip`, `*.resources` from history would reclaim **≈ 1.5–1.6 GB** → projected clone **< 200 MB**, matching the issue's Phase-2 estimate.

**Phase 2 is DESTRUCTIVE and gated on jsboige GO** (rewrites all SHAs → backup + coordinated force-push + all machines re-clone + old PR refs invalidated). Not actionable autonomously.

## Sources to PRESERVE (regenerable = False)

| At HEAD | Size | Nature |
|---|---:|---|
| `Cards/Packaging/*.ai/*.pdf/*.svg` (master box, FCPM, box designs) | **97 MB** | Illustrator/PDF design sources — not regenerable |
| `Generation/Sketch/argumentum.sketch` | **45 MB** | Sketch design source |
| `Cards/Fallacies/Assets/*.png` | 76 MB | Card art — pipeline + curated |

These are **Phase 3** candidates (Git LFS or external GDrive storage), again a jsboige decision. They must NOT be touched by a blind `filter-repo`.

## Note: Mindmap SVGs still tracked at HEAD (48 MB)

`Data/Mindmap/*.svg` (e.g. `Argumentum_Fallacies_MindMap_Fr_4.svg` 26 MB) are FreeMind-Batik output (regenerable) yet remain tracked. They were not covered by the Phase-1 untrack (PR #416). A small, safe follow-up could `git rm --cached` them — but they are byte-stable deliverables (#565) and some are referenced; leaving them is the conservative choice. Flagged, not actioned.

## Interim onboarding mitigation (non-destructive, actionable NOW)

The issue's original pain — *clone is very long on a new machine* — is **not blocked on Phase 2**. Configure/document a partial clone (keeps full history on demand):

```bash
# recommended: full history, blobs fetched lazily
git clone --filter=blob:none https://github.com/ArgumentumGames/Argumentum.git
# or fastest (no history):
git clone --depth=1 https://github.com/ArgumentumGames/Argumentum.git
```

No `partialclone.filter` default is currently set on the repo — new clones still fetch all 2 GB. Documenting the `--filter=blob:none` command in the README onboarding section (or setting a server-side default) unblocks new machines immediately, independent of the Phase-2 decision. **This is the one concrete, release-safe action this audit recommends.**

## Verdict

- **Phase 1**: ✅ DONE (#416 + #501). Close that section of the issue or mark it resolved.
- **Phase 2**: ⏸️ the only remaining weight-reducer; ~1.5 GB reclaimable; **jsboige-gated** (destructive).
- **Phase 3**: design sources (142 MB) → LFS/external; jsboige decision.
- **Interim**: document `--filter=blob:none` clone for onboarding (non-destructive).

Relates to #415, #416, #501, #134 (releases as the distributables home). Implements the "stale-dispatch" discipline: verify live state before acting on issue text.
106 changes: 106 additions & 0 deletions tools/git-weight-audit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""#415 repo-weight audit — read-only, reproducible.

Measures .git weight: pack size, top history blobs (deduped by path),
top HEAD files, and aggregates by zone with a regenerability flag.
0 write (read-only git plumbing). Run: python tools/git-weight-audit.py
"""
import subprocess, sys, re
from collections import defaultdict

TOP = 15 # per-listing limit

def git(*args):
r = subprocess.run(["git"]+list(args), capture_output=True, text=True, errors="replace")
return r.stdout

def human(n):
for u in ("B","KB","MB","GB"):
if n < 1024: return f"{n:.1f} {u}"
n /= 1024
return f"{n:.1f} TB"

# --- zones (path-prefix -> (name, regenerable, rationale)) ---
ZONES = [
(r"Generation/Converters/Argumentum.AssetConverter/Published/", "Published/ .NET builds", True, "dotnet publish (regenerable)"),
(r"Cartes/Generation/Converters/.*Published/", "Legacy Cartes/ builds", True, "legacy tree, deleted at HEAD"),
(r"DNNPlatform/Portals/.*/Downloads/.*\.zip", "DNN Downloads zips", True, "pipeline output (Print&Play)"),
(r"DNNPlatform/.*/(Install|ExtensionPackages)/.*\.resources", "2sxc/DNN .resources", True, "re-downloadable install pkgs"),
(r"Generation/Sketch/.*\.sketch", "Sketch design source", False, "DESIGN SOURCE — preserve/LFS"),
(r"Cards/Packaging/", "Packaging design", False, "DESIGN SOURCE — preserve/LFS"),
(r"Cards/Fallacies/Assets/.*\.(png|jpg)", "Card PNG assets", "part", "pipeline + curated"),
(r"Data/Mindmap/.*\.svg", "Mindmap SVGs", True, "FreeMind Batik (regenerable)"),
]

def zone_of(path):
for pat, name, regen, _ in ZONES:
if re.search(pat, path): return name, regen
return "other (text/code)", None

def main():
print("="*70); print(" #415 REPO-WEIGHT AUDIT"); print("="*70)
# 1. count-objects
co = git("count-objects","-vH")
pack = {l.split(": ")[0]: l.split(": ")[1] for l in co.splitlines() if ": " in l}
print(f"\n[pack] size-pack = {pack.get('size-pack','?')} | in-pack = {pack.get('in-pack','?')} | packs = {pack.get('packs','?')}")

# 2. history blobs (deduped by path, keep largest)
raw = git("rev-list","--objects","--all")
batch_in = "\n".join(l for l in raw.splitlines() if l)
# batch-check needs object list; feed rev-list objects via stdin
objs = subprocess.run(["git","cat-file","--batch-check=%(objecttype) %(objectname) %(objectsize) %(rest)"],
input=raw, capture_output=True, text=True, errors="replace").stdout
hist_path_max = {} # path -> max bytes
for line in objs.splitlines():
p = line.split(" ",3)
if len(p)<4 or p[0]!="blob": continue
try: sz=int(p[2])
except: continue
path=p[3].strip()
if path and sz>hist_path_max.get(path,0): hist_path_max[path]=sz

# 3. HEAD files
head = git("ls-tree","-r","-l","HEAD")
head_files=[] # (bytes, path)
for line in head.splitlines():
p=line.split(None,4)
if len(p)<5: continue
try: sz=int(p[3])
except: continue
head_files.append((sz, p[4].strip()))

# 4. zone aggregation
print(f"\n[top {TOP} history blobs (deduped by path, largest version)]")
for path,sz in sorted(hist_path_max.items(), key=lambda x:-x[1])[:TOP]:
print(f" {human(sz):>11} {path}")
print(f"\n[top {TOP} HEAD files]")
for sz,path in sorted(head_files, reverse=True)[:TOP]:
print(f" {human(sz):>11} {path}")

# zone totals: history (sum of unique-path max) + HEAD (sum of present)
zh = defaultdict(int); zh_n=defaultdict(int) # history bytes + count
zhead=defaultdict(int); zhead_n=defaultdict(int)
regen={}
for path,sz in hist_path_max.items():
z,r = zone_of(path); zh[z]+=sz; zh_n[z]+=1; regen[z]=r
for sz,path in head_files:
z,r = zone_of(path); zhead[z]+=sz; zhead_n[z]+=1; regen[z]=r

print("\n[zone aggregation]")
print(f" {'zone':30s} {'history':>12} {'(files)':>9} {'at HEAD':>12} {'(files)':>9} regenerable")
for z in sorted(set(list(zh)+list(zhead)), key=lambda z:-zh.get(z,0)):
rh = zh.get(z,0); rhn=zh_n.get(z,0); rhd=zhead.get(z,0); rhdn=zhead_n.get(z,0)
print(f" {z:30s} {human(rh):>12} {rhn:>9} {human(rhd):>12} {rhdn:>9} {regen.get(z,'?')}")

total_hist = sum(zh.values()); total_head=sum(sz for sz,_ in head_files)
print(f"\n[totals] history unique-path bytes (largest ver) ≈ {human(total_hist)} | HEAD checkout ≈ {human(total_head)}")

# proof-of-preservation summary
print("\n[proof-of-preservation — regenerable zones]")
for pat,name,regen,why in ZONES:
if regen is True:
n=zh_n.get(name,0)+zhead_n.get(name,0)
print(f" {name:30s} {n:4d} files -> {why}")

if __name__=="__main__":
main()
Loading