diff --git a/pmoves/docs/PMOVESCHIT/CALIBRATION_GUIDE.md b/pmoves/docs/PMOVESCHIT/CALIBRATION_GUIDE.md new file mode 100644 index 0000000000..3488cde0bb --- /dev/null +++ b/pmoves/docs/PMOVESCHIT/CALIBRATION_GUIDE.md @@ -0,0 +1,435 @@ +# CHIT Calibration Guide + +**Layer:** L1 Protocol / L3 Applied +**Status:** Current +**Last Updated:** 2026-03-11 + +> Procedures for calibrating CHIT encoding/decoding quality using KL divergence, JS divergence, Wasserstein distance, and coverage metrics. Includes codebook sandboxing, parameter tuning, and troubleshooting. + +--- + +## Table of Contents + +1. [Overview](#overview) +2. [Calibration Metrics](#calibration-metrics) +3. [Baseline Calibration](#baseline-calibration) +4. [Codebook Sandboxing](#codebook-sandboxing) +5. [Parameter Tuning](#parameter-tuning) +6. [Monitoring & Alerts](#monitoring--alerts) +7. [Troubleshooting](#troubleshooting) +8. [Cross-References](#cross-references) + +--- + +## Overview + +Calibration ensures that CGP packets accurately represent their source content and can be faithfully decoded. The key question is: **does the decoded output match the original?** + +### When to Calibrate + +| Trigger | Action | +|---------|--------| +| New embedding model deployed | Full recalibration | +| K or bins changed | Spectrum recalibration | +| Corpus significantly updated | Codebook refresh | +| Metrics drift detected | Incremental tuning | +| New content modality added | Modality-specific calibration | + +### Calibration Pipeline + +``` +Encode → Decode → Compare → Metrics → Adjust → Re-encode + ↑ │ + └──────────────────────────────────────────────────┘ +``` + +--- + +## Calibration Metrics + +### KL Divergence (Kullback-Leibler) + +Measures how the decoded spectrum diverges from the encoded spectrum: + +``` +KL(P || Q) = sum(P[i] * log(P[i] / Q[i])) + +where: + P = encoded spectrum (target) + Q = decoded spectrum (reconstructed) +``` + +| KL Value | Quality | Action | +|----------|---------|--------| +| < 0.1 | Excellent | No action needed | +| 0.1 - 0.3 | Good | Acceptable for production | +| 0.3 - 0.5 | Fair | Consider tuning | +| > 0.5 | Poor | Recalibration required | + +**Important**: KL divergence is asymmetric. Always compute KL(encoded || decoded), not the reverse. + +### JS Divergence (Jensen-Shannon) + +Symmetric version of KL, bounded [0, 1]: + +``` +JS(P, Q) = 0.5 * KL(P || M) + 0.5 * KL(Q || M) +where M = 0.5 * (P + Q) +``` + +| JS Value | Quality | Action | +|----------|---------|--------| +| < 0.05 | Excellent | No action | +| 0.05 - 0.15 | Good | Acceptable | +| 0.15 - 0.3 | Fair | Tune parameters | +| > 0.3 | Poor | Recalibrate | + +### Wasserstein-1D (Earth Mover's Distance) + +Measures the minimum "work" to transform one distribution into another: + +``` +W1(P, Q) = integral(|CDF_P(x) - CDF_Q(x)| dx) +``` + +More robust to small shifts than KL/JS. Useful when spectra are close but slightly offset. + +| W1 Value | Quality | +|----------|---------| +| < 0.1 | Excellent | +| 0.1 - 0.2 | Good | +| > 0.2 | Needs attention | + +### Coverage + +Fraction of original content units successfully recovered by the decoder: + +``` +coverage = |decoded_units intersect original_units| / |original_units| +``` + +| Coverage | Quality | +|----------|---------| +| > 0.9 | Excellent | +| 0.8 - 0.9 | Good | +| 0.6 - 0.8 | Fair (geometry-only mode expected) | +| < 0.6 | Poor | + +--- + +## Baseline Calibration + +### Step 1: Prepare Calibration Corpus + +```python +# Use a representative corpus (1000-5000 units) +corpus_texts = load_corpus("calibration_set.jsonl") +corpus_vecs = model.encode(corpus_texts, normalize_embeddings=True) +``` + +Requirements: +- **Representative**: Same domain/style as production content +- **Diverse**: Covers the full semantic range expected +- **Labeled**: Ground-truth topics/categories known (if possible) +- **Size**: 1000-5000 units for reliable statistics + +### Step 2: Encode + +```python +from pmoves.tools.chr import ConstellationHarvest + +chr = ConstellationHarvest(K=8, bins=8) +U = chr.optimize_anchors(corpus_vecs) +p = chr.compute_assignments(corpus_vecs, U) +spectra = chr.compute_spectra(corpus_vecs, U, p, bins=8) +cgp = build_cgp(U, spectra, corpus_vecs, corpus_texts, meta={...}) +``` + +### Step 3: Decode + +```python +from pmoves.tools.chit.chit_decoder import decode_geometry + +decoded = decode_geometry( + cgp=cgp, + corpus_texts=corpus_texts, + corpus_vecs=corpus_vecs, + top_k=10 # candidates per constellation +) +``` + +### Step 4: Measure + +```python +from pmoves.tools.chit.chit_decoder import compute_metrics + +metrics = compute_metrics( + cgp=cgp, + decoded=decoded, + corpus_texts=corpus_texts, + corpus_vecs=corpus_vecs +) + +print(f"Mean KL: {metrics['mean']['KL']:.4f}") +print(f"Mean JS: {metrics['mean']['JS']:.4f}") +print(f"Mean W1: {metrics['mean']['W1']:.4f}") +print(f"Mean Coverage: {metrics['mean']['coverage']:.3f}") +print(f"Per-constellation KL: {metrics['per_constellation']['KL']}") +``` + +### Step 5: Record Baseline + +```json +{ + "calibration_id": "cal-2026-03-11", + "backend": "sentence-transformers/all-MiniLM-L6-v2", + "K": 8, + "bins": 8, + "corpus_size": 3000, + "metrics": { + "mean_KL": 0.18, + "mean_JS": 0.08, + "mean_W1": 0.12, + "mean_coverage": 0.87 + }, + "per_constellation": [ + {"id": "const_0_0", "KL": 0.12, "coverage": 0.92}, + {"id": "const_0_1", "KL": 0.25, "coverage": 0.81} + ] +} +``` + +--- + +## Codebook Sandboxing + +The **Universal Codebook Property** requires that encoder and decoder share the same embedding model and corpus. When the corpus changes, the codebook must be refreshed. + +### What is a Codebook? + +A codebook is the decoder's reference corpus — the set of (text, embedding) pairs used to reconstruct content from geometric coordinates. + +### Sandboxing Strategy + +``` +Production Codebook + └── Sandbox A (new content added) + └── Sandbox B (different K/bins) + └── Sandbox C (different model) +``` + +### Creating a Sandbox + +```python +# Fork the production codebook +sandbox = ProductionCodebook.fork("sandbox-A") + +# Add new content +sandbox.add_texts(new_texts, model) + +# Calibrate against known CGPs +metrics = sandbox.calibrate(reference_cgps) + +# Promote if metrics pass +if metrics["mean_KL"] < 0.3 and metrics["mean_coverage"] > 0.8: + sandbox.promote_to_production() +``` + +### Codebook Refresh Schedule + +| Trigger | Frequency | Impact | +|---------|-----------|--------| +| New content ingested | Weekly | Low (incremental add) | +| Model version change | On deploy | High (full rebuild) | +| K/bins change | On config change | Medium (re-encode) | +| Quality drift > 20% | On detection | High (investigate root cause) | + +--- + +## Parameter Tuning + +### K (Number of Constellations) + +| Symptom | Diagnosis | Fix | +|---------|-----------|-----| +| High KL on all constellations | K too low, mixed topics | Increase K | +| Many near-empty constellations | K too high | Decrease K | +| One constellation has most points | Unbalanced assignment | Adjust tau or increase K | + +**Rule of thumb**: `K ~ sqrt(N / 10)` where N = number of content units. + +### bins (Spectrum Resolution) + +| Symptom | Diagnosis | Fix | +|---------|-----------|-----| +| Spectra look flat/uniform | bins too low | Increase bins | +| Spectra look noisy/spiky | bins too high | Decrease bins | +| W1 high but KL low | Binning artifacts | Try adjacent bin counts | + +**Rule of thumb**: 8 bins works for most use cases. Use 5-6 for < 100 units, 10-12 for > 5000 units. + +### tau (Temperature) + +| Symptom | Diagnosis | Fix | +|---------|-----------|-----| +| All points assigned to one constellation | tau too low | Increase tau | +| Assignments nearly uniform | tau too high | Decrease tau | +| Good separation but some outliers | tau is fine | Consider removing outliers | + +**Range**: 0.01 (hard assignment) to 1.0 (soft/uniform). Default: 0.1. + +### Zeta Filter Parameters + +| Parameter | Effect of Increase | Effect of Decrease | +|-----------|-------------------|-------------------| +| `numZeros` | More harmonics, smoother filtering | Fewer harmonics, sharper filtering | +| `decayFactor` | Higher zeros contribute more | Focus on lowest zeros | + +**Recommended**: numZeros=10, decayFactor=0.9 for general text. Use numZeros=5 for short documents. + +--- + +## Monitoring & Alerts + +### Prometheus Metrics + +Services publishing CGPs should expose: + +```python +# In your service metrics +cgp_kl_divergence = Histogram( + 'cgp_kl_divergence', + 'KL divergence of CGP encoding', + buckets=[0.05, 0.1, 0.2, 0.3, 0.5, 1.0] +) + +cgp_coverage = Histogram( + 'cgp_coverage', + 'Coverage of CGP decoding', + buckets=[0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0] +) +``` + +### Alert Thresholds + +```yaml +# Grafana alert rules +- alert: CGPQualityDegraded + expr: histogram_quantile(0.5, cgp_kl_divergence_bucket) > 0.5 + for: 1h + labels: + severity: warning + annotations: + summary: "CGP encoding quality degraded (median KL > 0.5)" + +- alert: CGPCoverageLow + expr: histogram_quantile(0.5, cgp_coverage_bucket) < 0.7 + for: 1h + labels: + severity: critical + annotations: + summary: "CGP decoding coverage below 70%" +``` + +### NATS Telemetry + +Calibration results are published to `geometry.cgp.calibration.v1`: + +```json +{ + "calibration_id": "cal-2026-03-11", + "timestamp": "2026-03-11T12:00:00Z", + "metrics": { + "mean_KL": 0.18, + "mean_coverage": 0.87 + }, + "status": "healthy" +} +``` + +EvoSwarm consumes this telemetry to drive parameter evolution. + +--- + +## Troubleshooting + +### High KL Divergence (> 0.5) + +**Causes:** +1. **Model mismatch**: Encoder and decoder using different embedding models +2. **Corpus drift**: Production content differs significantly from calibration corpus +3. **Parameter mismatch**: K/bins changed without recalibration +4. **Anchor collapse**: Multiple anchors converged to similar directions + +**Diagnosis:** +```python +# Check per-constellation KL +for k, kl in enumerate(metrics["per_constellation"]["KL"]): + if kl > 0.5: + print(f"Constellation {k}: KL={kl:.4f} — investigate anchor quality") + +# Check anchor similarity +for i in range(K): + for j in range(i+1, K): + sim = np.dot(U[i], U[j]) + if sim > 0.9: + print(f"Anchors {i},{j} too similar (cos={sim:.3f})") +``` + +### Low Coverage (< 0.7) + +**Causes:** +1. **Missing corpus items**: New content not in decoder's codebook +2. **Projection bounds too tight**: `radial_minmax` excludes valid projections +3. **top_k too low**: Not enough candidates considered +4. **Domain shift**: Content from a new domain not covered by corpus + +**Fix:** +```python +# Increase top_k +decoded = decode_geometry(cgp, corpus_texts, corpus_vecs, top_k=50) + +# Check for uncovered regions +uncovered = set(range(len(corpus_texts))) - set(decoded_indices) +print(f"Uncovered units: {len(uncovered)} / {len(corpus_texts)}") +``` + +### Entropy Trajectory Not Converging + +If `Hg_traj` doesn't decrease or `Hs_traj` collapses to near-zero: + +```python +# Check iteration count +if len(Hg_traj) < 50: + print("Increase max_iter (current may be insufficient)") + +# Check beta regularization +if Hs_traj[-1] < 0.01: + print("Increase beta to prevent anchor collapse") +``` + +### Zeta Filter Distortion + +If `spectrum_zeta` looks very different from `spectrum`: + +```python +# Compare filtered vs raw +for k in range(K): + delta = np.abs(np.array(spectra_zeta[k]) - np.array(spectra[k])) + if delta.max() > 0.2: + print(f"Constellation {k}: zeta distortion too high, reduce numZeros") +``` + +--- + +## Cross-References + +- [MATH_PIPELINE_WALKTHROUGH.md](MATH_PIPELINE_WALKTHROUGH.md) — Complete pipeline narrative +- [CGP_ENCODING_REFERENCE.md](CGP_ENCODING_REFERENCE.md) — Field-by-field packet construction +- [CGP_v1.0_SPECIFICATION.md](CGP_v1.0_SPECIFICATION.md) — Official wire format spec +- [EVOSWARM_OPERATIONS_GUIDE.md](../EVOSWARM_OPERATIONS_GUIDE.md) — Parameter pack optimization +- [CHIT Tools Catalog](../CHIT_TOOLS_CATALOG.md) — Python tooling reference + +--- + +*This document is a living artifact tracked by [CHIT_CHANGE_TRACKER.md](../CHIT_CHANGE_TRACKER.md).* diff --git a/pmoves/docs/PMOVESCHIT/CGP_ENCODING_REFERENCE.md b/pmoves/docs/PMOVESCHIT/CGP_ENCODING_REFERENCE.md new file mode 100644 index 0000000000..e2b698b047 --- /dev/null +++ b/pmoves/docs/PMOVESCHIT/CGP_ENCODING_REFERENCE.md @@ -0,0 +1,653 @@ +# CGP Encoding Reference + +**Layer:** L1 Protocol +**Status:** Current +**Last Updated:** 2026-03-11 + +> Field-by-field reference for constructing CHIT Geometry Packets (CGP v1.0). Includes JSON schema examples, validation rules, and construction patterns for each section of the packet. + +--- + +## Table of Contents + +1. [Packet Skeleton](#packet-skeleton) +2. [Root Fields](#root-fields) +3. [Meta Block](#meta-block) +4. [Signature Block](#signature-block) +5. [Super-Nodes](#super-nodes) +6. [Constellations](#constellations) +7. [Points](#points) +8. [MACA Consensus Block](#maca-consensus-block) +9. [Anchor Encryption Block](#anchor-encryption-block) +10. [Hyperbolic Encoding Block](#hyperbolic-encoding-block) +11. [Schema Versions](#schema-versions) +12. [Validation Checklist](#validation-checklist) +13. [Construction Examples](#construction-examples) +14. [JSON Schema Reference](#json-schema-reference) + +--- + +## Packet Skeleton + +Minimal valid CGP v1.0: + +```json +{ + "spec": "chit.cgp.v1.0", + "meta": { + "source": "text", + "units_mode": "sentences", + "K": 4, + "bins": 8, + "backend": "sentence-transformers/all-MiniLM-L6-v2", + "created_at": "2026-03-11T00:00:00Z" + }, + "super_nodes": [ + { + "id": "super_0", + "label": "Mode 0", + "constellations": [ + { + "id": "const_0_0", + "anchor": [0.1, -0.2, 0.3], + "radial_minmax": [-0.5, 0.9], + "spectrum": [0.25, 0.25, 0.25, 0.25, 0.0, 0.0, 0.0, 0.0] + } + ] + } + ] +} +``` + +--- + +## Root Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `spec` | string | **Yes** | Protocol version identifier | +| `meta` | object | **Yes** | Encoding metadata | +| `sig` | object | No | HMAC signature block | +| `super_nodes` | array | **Yes** | Top-level geometric clusters | + +### `spec` Values + +| Value | Status | Notes | +|-------|--------|-------| +| `chit.cgp.v1.0` | **Current** | Full production spec | +| `chit.cgp.v0.2` | Stable | Attribution + Merkle, no zeta | +| `chit.cgp.v0.1` | Legacy | Basic super_nodes only | + +**Legacy aliases** (accepted but deprecated): +- `cgp.v1` maps to `chit.cgp.v1.0` +- `geometry.cgp.v1` maps to `chit.cgp.v1.0` + +--- + +## Meta Block + +```json +{ + "meta": { + "source": "text", + "units_mode": "sentences", + "K": 8, + "bins": 8, + "backend": "sentence-transformers/all-MiniLM-L6-v2", + "mhep": 72.3, + "Hg_traj": [0.98, 0.92, 0.85, 0.77], + "Hs_traj": [1.22, 1.15, 1.08, 1.01], + "created_at": "2026-03-11T12:00:00Z", + "encoder_version": "1.0.0", + "zeta_filtering": true + } +} +``` + +### Field Details + +| Field | Type | Required | Validation | Description | +|-------|------|----------|------------|-------------| +| `source` | enum | **Yes** | `docx\|text\|latent\|image\|audio\|video` | Content modality | +| `units_mode` | enum | **Yes** | `paragraphs\|sentences\|tokens\|frames\|samples` | Segmentation granularity | +| `K` | integer | **Yes** | 4-16 | Number of constellations | +| `bins` | integer | **Yes** | 5-12 | Spectrum histogram resolution | +| `backend` | string | **Yes** | — | Embedding model identifier | +| `mhep` | float | No | > 0 | Multi-scale Hyperbolic Entropy Product | +| `Hg_traj` | float[] | No | Monotonically decreasing | Global entropy trajectory | +| `Hs_traj` | float[] | No | Should stabilize | Slab entropy trajectory | +| `created_at` | string | No | ISO 8601 | Creation timestamp | +| `encoder_version` | string | No | semver | Encoder software version | +| `zeta_filtering` | boolean | No | — | Whether zeta filter was applied | + +### Source-Backend Pairings + +| Source | Recommended Backend | Dimensions | +|--------|-------------------|-----------| +| `text`, `docx` | `sentence-transformers/all-MiniLM-L6-v2` | 384 | +| `image` | `clip-ViT-B-32` | 512 | +| `audio` | `whisper-base` + `all-MiniLM-L6-v2` | 384 | +| `video` | `clip-ViT-B-32` + temporal pooling | 512 | +| `latent` | (pre-embedded, any) | varies | + +--- + +## Signature Block + +```json +{ + "sig": { + "alg": "HMAC-SHA256", + "kid": "9f86d081884c7d65", + "ts": 1739001600, + "hmac": "rTfzmOEHraWrVGjJW+tmEftsEXjl08dJmoi/gDCQfzo=" + } +} +``` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `alg` | string | **Yes** | Always `"HMAC-SHA256"` | +| `kid` | string | **Yes** | Key identifier: `sha256(passphrase)[:16]` | +| `ts` | integer | No | Unix timestamp of signing | +| `hmac` | string | **Yes** | Base64-encoded HMAC digest | + +### Signing Process + +1. Deep-copy the CGP document +2. Remove the `sig` field if present +3. Canonicalize: `json.dumps(doc, sort_keys=True, separators=(",", ":"))` +4. Compute: `hmac.new(passphrase.encode(), canonical.encode(), sha256).digest()` +5. Base64-encode the digest +6. Attach `sig` block + +### Verification Process + +1. Extract and save `sig` block +2. Remove `sig` from document +3. Re-canonicalize with same parameters +4. Re-compute HMAC with known passphrase +5. Constant-time compare with `sig.hmac` + +--- + +## Super-Nodes + +Super-nodes are the top-level geometric clusters. A CGP has 1 to K super-nodes. + +```json +{ + "id": "super_0", + "label": "Resonant Mode 0", + "summary": "Economic transactions and token distribution patterns", + "x": -212.3, + "y": 148.1, + "r": 260.0, + "hyperbolic": { + "poincare": [0.1, -0.05], + "curvature": -1.0 + }, + "constellations": [ ... ] +} +``` + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `id` | string | **Yes** | Unique identifier (e.g., `"super_0"`) | +| `label` | string | No | Human-readable label | +| `summary` | string | No | Descriptive summary of the cluster | +| `x` | float | No | 2D visualization X coordinate | +| `y` | float | No | 2D visualization Y coordinate | +| `r` | float | No | Visualization radius | +| `hyperbolic` | object | No | Poincare disk encoding (see [Hyperbolic Block](#hyperbolic-encoding-block)) | +| `constellations` | array | **Yes** | Child constellation clusters | + +### ID Conventions + +- Super-nodes: `super_{index}` (e.g., `super_0`, `super_1`) +- Constellations: `const_{super}_{index}` (e.g., `const_0_0`, `const_0_1`) +- Points: `pt_{super}_{const}_{index}` (e.g., `pt_0_0_0`) + +--- + +## Constellations + +Constellations are the core geometric unit — each represents a semantically coherent cluster defined by an anchor vector and energy spectrum. + +```json +{ + "id": "const_0_0", + "label": "Token Distribution", + "summary": "Weekly GroToken allocation patterns", + "anchor": [0.012, -0.31, 0.82, 0.15, -0.07, 0.44, 0.11, -0.22], + "radial_minmax": [-0.45, 0.93], + "spectrum": [0.08, 0.11, 0.15, 0.22, 0.18, 0.12, 0.08, 0.06], + "spectrum_zeta": [0.09, 0.12, 0.16, 0.23, 0.17, 0.11, 0.07, 0.05], + "points": [ ... ], + "maca_consensus": { ... }, + "anchor_enc": { ... } +} +``` + +| Field | Type | Required | Validation | Description | +|-------|------|----------|------------|-------------| +| `id` | string | **Yes** | Unique within CGP | Constellation identifier | +| `label` | string | No | — | Human-readable label | +| `summary` | string | No | — | Topic keywords or description | +| `anchor` | float[] | **Yes*** | L2 norm = 1.0 | Unit vector defining the cluster direction | +| `radial_minmax` | [float, float] | **Yes** | min < max | Projection bounds [min, max] | +| `spectrum` | float[] | **Yes** | Length = meta.bins, sum = 1.0 | Energy distribution histogram | +| `spectrum_zeta` | float[] | No | Length = meta.bins | Zeta-filtered spectrum | +| `points` | array | No | — | Individual data points | +| `maca_consensus` | object | No | — | Multi-agent consensus data | +| `anchor_enc` | object | No | — | Encrypted anchor (replaces `anchor`) | + +*`anchor` is required unless `anchor_enc` is provided. + +### Anchor Validation + +```python +import numpy as np + +anchor = np.array(constellation["anchor"]) +assert abs(np.linalg.norm(anchor) - 1.0) < 1e-6, "Anchor must be unit vector" +``` + +### Spectrum Validation + +```python +spectrum = constellation["spectrum"] +assert len(spectrum) == cgp["meta"]["bins"], "Spectrum length must match meta.bins" +assert abs(sum(spectrum) - 1.0) < 1e-6, "Spectrum must sum to 1.0" +assert all(v >= 0 for v in spectrum), "Spectrum values must be non-negative" +``` + +--- + +## Points + +Individual content units within a constellation. + +```json +{ + "id": "pt_0_0_0", + "x": 13.4, + "y": -8.2, + "proj": 0.83, + "conf": 0.94, + "text": "Weekly token distribution shows stable growth patterns", + "text_b64": "V2Vla2x5IHRva2VuIGRpc3Ry...", + "char_len": 55, + "word_count": 8, + "contributor_id": "agent-zero", + "merkle_proof": ["a1b2c3...", "d4e5f6..."], + "weight": 0.85 +} +``` + +| Field | Type | Required | Validation | Description | +|-------|------|----------|------------|-------------| +| `id` | string | **Yes** | Unique within CGP | Point identifier | +| `x` | float | No | — | 2D visualization X | +| `y` | float | No | — | 2D visualization Y | +| `proj` | float | Recommended | Within `radial_minmax` | Projection onto anchor | +| `conf` | float | Recommended | [0.0, 1.0] | Assignment confidence | +| `text` | string | No | — | Original text content | +| `text_b64` | string | No | Valid base64 | Base64-encoded text | +| `char_len` | integer | No | > 0 | Character count | +| `word_count` | integer | No | > 0 | Word count | +| `contributor_id` | string | No | — | Who contributed this unit | +| `merkle_proof` | string[] | No | Valid hash chain | Merkle proof path | +| `weight` | float | No | [0.0, 1.0] | Dirichlet attribution weight | + +### Projection Validation + +```python +proj = point["proj"] +rmin, rmax = constellation["radial_minmax"] +assert rmin <= proj <= rmax, f"Projection {proj} outside [{rmin}, {rmax}]" +``` + +### Text Encoding + +Either `text` or `text_b64` (or both) may be present: +- `text`: UTF-8 string, human-readable +- `text_b64`: Base64-encoded UTF-8 bytes, preferred for binary-safe transport +- If both present, `text_b64` takes precedence + +### Merkle Proof Structure + +The `merkle_proof` array contains sibling hashes from leaf to root: + +``` +proof[0] = sibling of leaf +proof[1] = sibling of parent +... +proof[n-1] = sibling of root's child +``` + +Verification: iteratively hash (leaf, proof[0]), then (result, proof[1]), etc. Final result should equal the constellation's merkle_root. + +--- + +## MACA Consensus Block + +Multi-Agent Consensus via Aggregation — records how multiple agents agreed on constellation assignment. + +```json +{ + "maca_consensus": { + "entropy_delta": 0.34, + "votes": [3, 4, 2], + "confidence": 0.85 + } +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `entropy_delta` | float | Entropy change from consensus (should be negative = more agreement) | +| `votes` | integer[] | Vote counts per round | +| `confidence` | float | Final consensus confidence [0, 1] | + +### MACA Process + +1. Multiple agents independently assign content units to constellations +2. Votes are aggregated across rounds +3. Entropy delta measures information gain from consensus +4. Confidence = fraction of agents agreeing on final assignment + +--- + +## Anchor Encryption Block + +When confidentiality is required, the `anchor` field is replaced with `anchor_enc`: + +```json +{ + "anchor_enc": { + "alg": "AES-GCM", + "iv": "YmFzZTY0X2l2X2hlcmU=", + "salt": "YmFzZTY0X3NhbHRfaGVyZQ==", + "ct": "YmFzZTY0X2NpcGhlcnRleHQ=" + } +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `alg` | string | Always `"AES-GCM"` | +| `iv` | string | Base64-encoded initialization vector (12 bytes) | +| `salt` | string | Base64-encoded PBKDF2 salt (16 bytes) | +| `ct` | string | Base64-encoded ciphertext (anchor + 16-byte auth tag) | + +### Key Derivation + +```python +from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC +from cryptography.hazmat.primitives import hashes + +kdf = PBKDF2HMAC( + algorithm=hashes.SHA256(), + length=32, + salt=salt_bytes, + iterations=100_000 +) +key = kdf.derive(passphrase.encode()) +``` + +### Encryption + +```python +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + +aesgcm = AESGCM(key) +ciphertext = aesgcm.encrypt(iv, anchor_json_bytes, None) +``` + +### Decryption + +```python +plaintext = aesgcm.decrypt(iv, ciphertext, None) +anchor = json.loads(plaintext) +``` + +--- + +## Hyperbolic Encoding Block + +Poincare disk coordinates for hierarchical positioning. + +```json +{ + "hyperbolic": { + "poincare": [0.1, -0.05], + "curvature": -1.0 + } +} +``` + +| Field | Type | Validation | Description | +|-------|------|------------|-------------| +| `poincare` | [float, float] | `sqrt(x^2 + y^2) < 1.0` | Poincare disk coordinates | +| `curvature` | float | Always -1.0 | Sectional curvature | + +### Coordinate Constraints + +```python +x, y = hyperbolic["poincare"] +r = math.sqrt(x**2 + y**2) +assert r < 1.0, "Point must be inside the Poincare disk" +assert r < 0.95, "Recommended: stay below 0.95 for numerical stability" +``` + +--- + +## Schema Versions + +### v1.0 (Current Production) + +Full spec with all fields documented above. Key additions over v0.2: +- `spectrum_zeta` field +- `maca_consensus` block +- `hyperbolic` block +- `encoder_version` in meta +- `zeta_filtering` flag in meta + +JSON Schema: `pmoves/contracts/schemas/geometry/cgp.v1.schema.json` + +### v0.2 (Stable) + +Attribution weights and Merkle proofs. Missing: +- Zeta-filtered spectrum +- MACA consensus +- Hyperbolic encoding +- Encoder version tracking + +### v0.1 (Legacy) + +Basic super_nodes and constellations. Missing: +- Attribution (no weights, no Merkle proofs) +- Security (no signature block) +- Metadata (minimal meta block) + +### Migration Path + +``` +v0.1 → v0.2: Add points[].weight, points[].merkle_proof, sig block +v0.2 → v1.0: Add spectrum_zeta, maca_consensus, hyperbolic, meta extensions +``` + +--- + +## Validation Checklist + +Before publishing a CGP v1.0 packet: + +- [ ] `spec` is `"chit.cgp.v1.0"` +- [ ] `meta.source` is a valid enum value +- [ ] `meta.units_mode` is a valid enum value +- [ ] `meta.K` matches number of constellations across all super_nodes +- [ ] `meta.bins` matches length of all `spectrum` arrays +- [ ] All `anchor` vectors have L2 norm = 1.0 (tolerance: 1e-6) +- [ ] All `spectrum` arrays sum to 1.0 (tolerance: 1e-6) +- [ ] All `spectrum` values are non-negative +- [ ] All `points[].proj` values within their constellation's `radial_minmax` +- [ ] All `points[].conf` values in [0.0, 1.0] +- [ ] All IDs are unique within the packet +- [ ] `sig.hmac` is valid if signature block present +- [ ] `anchor_enc` replaces `anchor` (never both present) +- [ ] `hyperbolic.poincare` coordinates within unit disk (r < 1.0) +- [ ] `created_at` is valid ISO 8601 + +--- + +## Construction Examples + +### Python: Full CGP Construction + +> **Note:** This example illustrates the designed pipeline flow. `ConstellationHarvest` and +> `ZetaSpectralFilter` are planned modules — see the +> [pipeline walkthrough](MATH_PIPELINE_WALKTHROUGH.md) for current implementation status. + +```python +import json +import numpy as np +from datetime import datetime, timezone +from pmoves.tools.chr import ConstellationHarvest +from pmoves.tools.zeta_filter import ZetaSpectralFilter +from pmoves.tools.chit_security import sign_cgp + +# 1. Embed content +embeddings = model.encode(texts, normalize_embeddings=True) + +# 2. Run CHR +chr = ConstellationHarvest(K=8, bins=8) +U = chr.optimize_anchors(embeddings) +p = chr.compute_assignments(embeddings, U) +spectra = chr.compute_spectra(embeddings, U, p, bins=8) +Hg, Hs = chr.compute_entropy_trajectory(embeddings, U, p) + +# 3. Zeta filter +zeta = ZetaSpectralFilter(num_zeros=10) +spectra_zeta = [zeta.filter_spectrum(s) for s in spectra] + +# 4. Construct CGP +cgp = { + "spec": "chit.cgp.v1.0", + "meta": { + "source": "text", + "units_mode": "sentences", + "K": 8, + "bins": 8, + "backend": "sentence-transformers/all-MiniLM-L6-v2", + "mhep": float(Hg[-1] * Hs[-1]), + "Hg_traj": [float(h) for h in Hg], + "Hs_traj": [float(h) for h in Hs], + "created_at": datetime.now(timezone.utc).isoformat(), + "encoder_version": "1.0.0", + "zeta_filtering": True + }, + "super_nodes": [] +} + +for k in range(8): + assigned = np.where(p[:, k] > 0.5)[0] + projections = embeddings[assigned] @ U[k] + rmin, rmax = float(projections.min()), float(projections.max()) + + constellation = { + "id": f"const_0_{k}", + "anchor": U[k].tolist(), + "radial_minmax": [rmin, rmax], + "spectrum": spectra[k].tolist(), + "spectrum_zeta": spectra_zeta[k].tolist(), + "points": [ + { + "id": f"pt_0_{k}_{j}", + "proj": float(projections[j]), + "conf": float(p[assigned[j], k]), + "text": texts[assigned[j]] + } + for j in range(len(assigned)) + ] + } + + if k == 0: + cgp["super_nodes"].append({ + "id": "super_0", + "label": "Primary Mode", + "constellations": [constellation] + }) + else: + cgp["super_nodes"][0]["constellations"].append(constellation) + +# 5. Sign +cgp = sign_cgp(cgp, passphrase="shared-secret") +``` + +### TypeScript: Using createCHITSystem + +```typescript +import { createCHITSystem } from '@pmoves/chit'; + +const chit = createCHITSystem({ + dirichlet: { smoothingAlpha: 0.1, concentrationK: 1.0, decayHalfLife: 12 }, + hyperbolic: { curvature: -1, baseRadius: 0.3 }, + merkle: { strategy: 'per_week' }, + cgp: { namespace: 'pmoves.tokenism' }, + swarm: { optimizationTarget: 'gini_reduction' } +}); + +// Record actions +chit.attribution.recordAction({ + address: '0xABC...', + action: 'spending', + amount: 50, + week: 12, + category: 'groceries' +}); + +// Generate weekly CGP +const cgp = chit.generator.generateWeeklyCGP(weekData, chit.attribution); + +// Publish +await chit.publisher.publishWeeklyCGP(12, cgp, { gini: 0.42 }); +``` + +--- + +## JSON Schema Reference + +### Location + +``` +pmoves/contracts/schemas/geometry/ + cgp.v1.schema.json # CGP v1.0 full schema + cgp.v2.schema.json # CGP v2.0 (draft, extends v1.0) + swarm.meta.v1.schema.json # EvoSwarm parameter pack schema +``` + +### Usage + +```python +import jsonschema, json + +with open("pmoves/contracts/schemas/geometry/cgp.v1.schema.json") as f: + schema = json.load(f) + +jsonschema.validate(cgp, schema) # Raises ValidationError on failure +``` + +--- + +## Cross-References + +- [MATH_PIPELINE_WALKTHROUGH.md](MATH_PIPELINE_WALKTHROUGH.md) — End-to-end pipeline narrative +- [CALIBRATION_GUIDE.md](CALIBRATION_GUIDE.md) — Decoder calibration procedures +- [CGP_v1.0_SPECIFICATION.md](CGP_v1.0_SPECIFICATION.md) — Official wire format spec +- [GEOMETRY_BUS_INTEGRATION.md](GEOMETRY_BUS_INTEGRATION.md) — NATS transport guide + +--- + +*This document is a living artifact tracked by [CHIT_CHANGE_TRACKER.md](../CHIT_CHANGE_TRACKER.md).* diff --git a/pmoves/docs/PMOVESCHIT/MATH_PIPELINE_WALKTHROUGH.md b/pmoves/docs/PMOVESCHIT/MATH_PIPELINE_WALKTHROUGH.md new file mode 100644 index 0000000000..7cd9130519 --- /dev/null +++ b/pmoves/docs/PMOVESCHIT/MATH_PIPELINE_WALKTHROUGH.md @@ -0,0 +1,865 @@ +# CHIT Mathematical Pipeline Walkthrough + +**Layer:** L1 Protocol / L2 Conceptual +**Status:** Current +**Last Updated:** 2026-03-11 + +> End-to-end walkthrough of the CHIT mathematical encoding pipeline: from raw content through Dirichlet attribution, hyperbolic embedding, zeta spectral filtering, to CGP packet construction, transport, and decoding. + +--- + +## Table of Contents + +1. [Pipeline Overview](#pipeline-overview) +2. [Stage 1: Content Ingestion & Embedding](#stage-1-content-ingestion--embedding) +3. [Stage 2: Constellation Harvest Regularization (CHR)](#stage-2-constellation-harvest-regularization-chr) +4. [Stage 3: Dirichlet Attribution Weighting](#stage-3-dirichlet-attribution-weighting) +5. [Stage 4: Hyperbolic Geometry Encoding](#stage-4-hyperbolic-geometry-encoding) +6. [Stage 5: Zeta Spectral Filtering](#stage-5-zeta-spectral-filtering) +7. [Stage 6: CGP Packet Construction](#stage-6-cgp-packet-construction) +8. [Stage 7: Security (Signing & Encryption)](#stage-7-security-signing--encryption) +9. [Stage 8: Transport (GEOMETRY BUS)](#stage-8-transport-geometry-bus) +10. [Stage 9: Decoding & Reconstruction](#stage-9-decoding--reconstruction) +11. [Stage 10: Swarm Optimization Feedback](#stage-10-swarm-optimization-feedback) +12. [Complete Data Flow Diagram](#complete-data-flow-diagram) +13. [Cross-References](#cross-references) + +--- + +## Pipeline Overview + +The CHIT (Cymatic-Holographic Information Transfer) pipeline transforms arbitrary content into **boundary representations** — geometric packets called CGPs (CHIT Geometry Packets) — that encode meaning as constellations of anchors, spectra, and attribution proofs rather than raw token streams. + +### Why Geometry Instead of Tokens? + +| Property | Token Stream | CGP Geometry | +|----------|-------------|--------------| +| **Compression** | Linear with content length | Logarithmic (boundary representation) | +| **Translatability** | Language-specific | Universal (geometric invariants) | +| **Attribution** | Lost after generation | Merkle-proven per contributor | +| **Privacy** | Content visible | Anchors can be encrypted (AES-GCM) | +| **Decodability** | Direct | Codebook-reconstructible or exact | + +### The Five Mathematical Pillars + +| Pillar | Mathematical Basis | Role in Pipeline | +|--------|-------------------|------------------| +| **Dirichlet Distributions** | Bayesian conjugate priors | Fair contribution weighting | +| **Hyperbolic Geometry** | Poincare disk (K = -1) | Hierarchical spatial encoding | +| **Zeta Spectral Filtering** | Riemann zeta zeros (gamma_k) | Signal enhancement via prime resonance | +| **Merkle Proofs** | Hash-based verification trees | Tamper-proof attribution chains | +| **Swarm Optimization** | Evolutionary algorithms | Distributed parameter consensus | + +### Pipeline Stages at a Glance + +``` +Content → Embed → CHR → Dirichlet → Hyperbolic → Zeta → CGP → Sign → NATS → Decode + (1) (1) (2) (3) (4) (5) (6) (7) (8) (9) + ↑ + Swarm Feedback (10) +``` + +--- + +## Stage 1: Content Ingestion & Embedding + +### Input Types + +The pipeline accepts six content modalities: + +| Source | units_mode | Embedding Model | Example | +|--------|-----------|----------------|---------| +| `text` | `sentences` or `paragraphs` | all-MiniLM-L6-v2 | Documents, transcripts | +| `docx` | `paragraphs` | all-MiniLM-L6-v2 | Word documents | +| `image` | `frames` | CLIP ViT-B-32 | Photos, screenshots | +| `audio` | `samples` | Whisper + embedding | Podcasts, music | +| `video` | `frames` | CLIP + temporal | YouTube, streams | +| `latent` | `tokens` | Pre-embedded vectors | Agent reasoning traces | + +### Embedding Process + +```python +from sentence_transformers import SentenceTransformer + +model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2') + +# 1. Split content into units +units = split_into_sentences(document) # or paragraphs, frames, etc. + +# 2. Generate normalized embeddings +embeddings = model.encode( + units, + normalize_embeddings=True, # L2 norm = 1.0 (critical for projection) + show_progress_bar=True +) +# Shape: (N, 384) where N = number of units, 384 = embedding dimension +``` + +### Key Properties + +- **Normalization**: All embeddings are L2-normalized to unit vectors. This is critical because later stages (projection, spectrum computation) assume `||v|| = 1`. +- **Dimensionality**: MiniLM-L6-v2 produces 384-dimensional vectors. CLIP ViT-B-32 produces 512-dimensional vectors. +- **Determinism**: Same input text always produces the same embedding (no temperature/sampling). + +--- + +## Stage 2: Constellation Harvest Regularization (CHR) + +CHR is the core algorithm that discovers **constellations** — clusters of semantically related content units arranged around **anchor vectors** in embedding space. + +### What CHR Produces + +| Output | Shape | Description | +|--------|-------|-------------| +| **Anchors (U)** | (K, D) | K unit vectors in D-dimensional space | +| **Assignments (p)** | (N, K) | Soft assignment probabilities per unit | +| **Spectra** | (K, bins) | Energy distribution per constellation | +| **Entropy trajectories** | (iterations,) | Convergence tracking | + +### Algorithm Overview + +```python +from pmoves.tools.chr import ConstellationHarvest + +chr = ConstellationHarvest(K=8, bins=8) + +# Step 2a: Optimize anchors via alternating minimization +U = chr.optimize_anchors(embeddings) +# U[k] is the "centroid direction" of constellation k +# Each anchor is a unit vector (||U[k]|| = 1) + +# Step 2b: Compute soft assignments +p = chr.compute_assignments(embeddings, U) +# p[i][k] = probability that unit i belongs to constellation k +# Uses softmax over cosine similarities with temperature tau + +# Step 2c: Compute energy spectra +spectra = chr.compute_spectra(embeddings, U, p, bins=8) +# spectra[k] = histogram of projections onto anchor U[k] +# Each spectrum sums to 1.0 (probability distribution) + +# Step 2d: Track convergence +Hg_traj, Hs_traj = chr.compute_entropy_trajectory(embeddings, U, p) +# Hg = global entropy (should decrease toward convergence) +# Hs = slab entropy (per-constellation spread) +``` + +### Anchor Optimization + +Anchors are found by minimizing reconstruction error: + +``` +minimize sum_k sum_i p[i][k] * (1 - cos(embedding[i], U[k])) +subject to ||U[k]|| = 1 for all k +``` + +This is equivalent to **spherical k-means** with soft assignments. The alternating minimization: + +1. **E-step**: Fix anchors, update assignments via softmax +2. **M-step**: Fix assignments, update anchors via weighted mean + normalize + +### Spectrum Computation + +For each constellation k: + +1. Project all assigned units onto anchor: `proj[i] = dot(embedding[i], U[k])` +2. Scale projections to [0, 1] using `radial_minmax` +3. Bin into histogram with `bins` buckets +4. Normalize to sum = 1.0 + +The spectrum is the **energy fingerprint** of a constellation — it captures how the content distributes along the anchor direction. + +### Configuration Parameters + +| Parameter | Default | Range | Effect | +|-----------|---------|-------|--------| +| `K` | 8 | 4-16 | Number of constellations (too few = coarse, too many = noisy) | +| `bins` | 8 | 5-12 | Spectrum resolution (higher = more detail, more data) | +| `tau` | 0.1 | 0.01-1.0 | Temperature for soft assignment (lower = harder assignment) | +| `beta` | 0.01 | 0-0.1 | Regularization weight (prevents anchor collapse) | +| `max_iter` | 100 | 50-500 | Optimization iterations | + +### Convergence Metrics + +- **MHEP (Multi-scale Hyperbolic Entropy Product)**: `Hg * Hs` — lower is better +- **Hg trajectory**: Global entropy should monotonically decrease +- **Hs trajectory**: Slab entropy should stabilize (not collapse to 0) + +--- + +## Stage 3: Dirichlet Attribution Weighting + +When content has multiple contributors (agents, humans, services), Dirichlet distributions ensure **fair, non-zero attribution**. + +### Mathematical Foundation + +The Dirichlet distribution Dir(alpha) is the conjugate prior for the multinomial distribution: + +``` +weights ~ Dir(alpha_1, alpha_2, ..., alpha_K) + +Expected weight for contributor i: + E[w_i] = alpha_i / sum(alpha) + +Concentration parameter: + sum(alpha) controls spread: + - Low sum(alpha) → spiky (one contributor dominates) + - High sum(alpha) → uniform (equal attribution) +``` + +### Fairness Guarantee + +With `alpha_i >= 1` for all contributors, **no contributor receives zero weight**. This is the core fairness property — even minimal contributions get non-zero attribution. + +### Configuration + +```typescript +interface DirichletConfig { + smoothingAlpha: number; // 0.1 = spiky, 10 = uniform (default: 0.1) + concentrationK: number; // Overall concentration (default: 1.0) + decayHalfLife: number; // Weeks before contribution halves (default: 12) +} +``` + +### Attribution Flow + +``` +Raw Contributions → Alpha Assignment → Decay Application → Normalization → Weights + $50 alpha = 51 * exp(-rate*t) / sum(alpha) 0.32 +``` + +1. **Alpha Assignment**: `alpha_i = smoothingAlpha + (amount * concentrationK)` +2. **Temporal Decay**: `alpha_new = alpha_old * exp(-ln(2) / halfLife * weeks_inactive)` +3. **Normalization**: `weight_i = alpha_i / sum(all_alpha)` +4. **Merkle Proof**: Each weight gets a leaf hash for verification + +### Output Structure + +```typescript +interface ContributionWeight { + address: string; // Contributor identifier + weight: number; // [0, 1], all weights sum to 1 + rawContribution: number; // Pre-weighting amount + alphaComponent: number; // Individual alpha value + category: string; // Contract type (groceries, grotoken, etc.) +} +``` + +### Why Dirichlet? + +- **Closed-form Bayesian update**: New data updates alpha directly (no recomputation) +- **Conjugate prior**: Mathematically elegant for categorical/multinomial data +- **Fairness by construction**: Non-zero guarantee with alpha >= 1 +- **Interpretable**: Each alpha_i is "pseudo-count" of contributions + +--- + +## Stage 4: Hyperbolic Geometry Encoding + +Content hierarchies are encoded in the **Poincare disk model** — a 2D representation of hyperbolic space with curvature K = -1. + +### Why Hyperbolic Space? + +| Property | Euclidean (flat) | Hyperbolic (curved) | +|----------|-----------------|-------------------| +| **Area growth** | ~r^2 (polynomial) | ~e^r (exponential) | +| **Tree embedding** | O(sqrt(n)) distortion | O(log n) distortion | +| **Hierarchy encoding** | Poor (flat structure) | Natural (center = root) | +| **Volume at radius r** | Pi * r^2 | 2*Pi*(cosh(r) - 1) | + +Hyperbolic space is **exponentially more capacious** than Euclidean space. A tree with n nodes can be embedded with only O(log n) distortion, compared to O(sqrt(n)) in Euclidean space. This makes it ideal for encoding hierarchical content structures. + +### Poincare Disk Model + +The Poincare disk is the open unit disk {(x,y) : x^2 + y^2 < 1} with the metric: + +``` +ds^2 = 4 * (dx^2 + dy^2) / (1 - x^2 - y^2)^2 +``` + +Key properties: +- **Origin** (0, 0) = root/aggregate concept +- **Boundary** (unit circle) = infinity in hyperbolic space +- **Distance** grows logarithmically toward the boundary +- **Geodesics** are circular arcs orthogonal to the boundary + +### Distance Formula + +``` +d_hyp(u, v) = arcosh(1 + 2 * ||u - v||^2 / ((1 - ||u||^2) * (1 - ||v||^2))) +``` + +Near the boundary (||u|| -> 1), small Euclidean differences map to large hyperbolic distances. This is what gives the disk its exponential capacity. + +### Mobius Addition (Hyperbolic Translation) + +``` +u (+) v = ((1 + 2 + ||v||^2) * u + (1 - ||u||^2) * v) + / (1 + 2 + ||u||^2 * ||v||^2) +``` + +Note: Mobius addition is **non-commutative** (u (+) v != v (+) u). This reflects the curvature of the space. + +### Encoding Hierarchy + +``` +Layer 0 (origin, r ~ 0): Aggregate concepts (economy total, document summary) +Layer 1 (r ~ 0.3): Contract types / Super-nodes (GroToken, FoodUSD) +Layer 2 (r ~ 0.5-0.7): Constellations (topic clusters) +Layer 3 (r ~ 0.8-0.95): Individual points (transactions, sentences) +``` + +### Configuration + +```typescript +interface EncoderConfig { + curvature: number; // -1 (fixed for Poincare disk) + baseRadius: number; // 0.3 (first-level node placement) + radiusGrowth: number; // 1.3 (growth per level) + angularSpread: number; // Angular distribution for siblings + maxRadius: number; // 0.95 (must be < 1.0 for numerical stability) +} +``` + +### Output: Poincare Points + +```typescript +interface PoincarePoint { + x: number; // [-1, 1] + y: number; // [-1, 1] + radius: number; // sqrt(x^2 + y^2), must be < 1.0 + theta: number; // [0, 2*pi] + id?: string; + label?: string; +} +``` + +### CGP Integration + +Each super-node gets a `hyperbolic` field: + +```json +{ + "hyperbolic": { + "poincare": [0.1, -0.05], + "curvature": -1.0 + } +} +``` + +--- + +## Stage 5: Zeta Spectral Filtering + +Zeta filtering enhances meaningful spectral patterns using the **non-trivial zeros of the Riemann zeta function** as resonant frequencies. + +### Mathematical Basis + +The Riemann zeta function zeta(s) = sum(1/n^s, n=1..inf) has non-trivial zeros at s = 1/2 + i*gamma_k: + +| k | gamma_k (imaginary part) | +|---|-------------------------| +| 1 | 14.134725... | +| 2 | 21.022039... | +| 3 | 25.010857... | +| 4 | 30.424876... | +| 5 | 32.935061... | +| ... | ... | +| 10 | 49.773832... | +| 20 | 77.144840... | + +These zeros encode the **intrinsic frequencies of prime number distribution** — fundamental harmonic patterns in number theory. + +### Why Zeta Zeros? + +The connection between zeta zeros and information: prime numbers are the "atoms" of multiplicative number theory. Their distribution exhibits quasi-random patterns that zeta zeros characterize. By filtering signals through these frequencies, we: + +1. **Enhance patterns** that align with fundamental mathematical harmonics +2. **Suppress noise** that doesn't resonate with prime structure +3. **Achieve scale invariance** — zeta zeros work across all data granularities + +### Filter Mechanism + +```python +# For each spectrum bin at frequency f: +weight[k] = exp(-(f - gamma[k])^2 / (2 * sigma^2)) # Gaussian kernel + +# Weighted combination: +filtered_value = sum(weight[k] * input_spectrum[f] for k in 1..numZeros) +``` + +### Configuration + +```typescript +interface ZetaFilterConfig { + numZeros: number; // 1-20 zeros to use (default: 10) + decayFactor: number; // 0-1, higher zeros decay (default: 0.9) + normalizeOutput: boolean; // Normalize result to [0,1] (default: true) +} +``` + +### Output: Spectral Analysis + +```typescript +interface SpectralAnalysis { + filtered: number[]; // Zeta-weighted spectrum values + dominantIndex: number; // Peak frequency index + concentration: number; // Gini-like energy measure [0,1] + entropy: number; // Shannon entropy of filtered spectrum +} +``` + +### In the CGP + +Each constellation carries both raw and zeta-filtered spectra: + +```json +{ + "spectrum": [0.08, 0.11, 0.15, 0.22, 0.18, 0.12, 0.08, 0.06], + "spectrum_zeta": [0.09, 0.12, 0.16, 0.23, 0.17, 0.11, 0.07, 0.05] +} +``` + +The `spectrum_zeta` field is optional but recommended — it provides enhanced signal-to-noise for downstream consumers. + +--- + +## Stage 6: CGP Packet Construction + +All preceding stages converge into a single JSON document: the **CHIT Geometry Packet (CGP) v1.0**. + +### Complete Structure + +```json +{ + "spec": "chit.cgp.v1.0", + + "meta": { + "source": "text", + "units_mode": "sentences", + "K": 8, + "bins": 8, + "backend": "sentence-transformers/all-MiniLM-L6-v2", + "mhep": 72.3, + "Hg_traj": [0.98, 0.92, 0.85, 0.77], + "Hs_traj": [1.22, 1.15, 1.08, 1.01], + "created_at": "2026-03-11T12:00:00Z", + "encoder_version": "1.0.0", + "zeta_filtering": true + }, + + "sig": { ... }, + + "super_nodes": [ + { + "id": "super_0", + "label": "Resonant Mode 0", + "summary": "Semantic cluster description", + "x": -212.3, + "y": 148.1, + "r": 260.0, + "hyperbolic": { + "poincare": [0.1, -0.05], + "curvature": -1.0 + }, + "constellations": [ + { + "id": "const_0_0", + "label": "Constellation Label", + "summary": "Topic keywords", + "anchor": [0.012, -0.31, 0.82, "...", 0.15], + "radial_minmax": [-0.45, 0.93], + "spectrum": [0.08, 0.11, 0.15, 0.22, 0.18, 0.12, 0.08, 0.06], + "spectrum_zeta": [0.09, 0.12, 0.16, 0.23, 0.17, 0.11, 0.07, 0.05], + "points": [ + { + "id": "pt_0_0_0", + "x": 13.4, + "y": -8.2, + "proj": 0.83, + "conf": 0.94, + "text": "Original unit text...", + "text_b64": "SGVsbG8gd29ybGQ=", + "char_len": 127, + "word_count": 22, + "contributor_id": "agent-zero", + "merkle_proof": ["hash1", "hash2"], + "weight": 0.85 + } + ], + "maca_consensus": { + "entropy_delta": 0.34, + "votes": [3, 4, 2], + "confidence": 0.85 + } + } + ] + } + ] +} +``` + +### Field Requirements + +| Field | Required | Validation Rule | +|-------|----------|----------------| +| `spec` | Yes | Must be `"chit.cgp.v1.0"` | +| `meta.source` | Yes | One of: docx, text, latent, image, audio, video | +| `meta.units_mode` | Yes | One of: paragraphs, sentences, tokens, frames, samples | +| `meta.K` | Yes | Integer 4-16 | +| `meta.bins` | Yes | Integer 5-12 | +| `anchor` | Yes | Float array, L2 norm = 1.0 | +| `radial_minmax` | Yes | [min, max] projection bounds | +| `spectrum` | Yes | Float array, sums to 1.0 | +| `points[].proj` | Recommended | Within radial_minmax bounds | +| `points[].conf` | Recommended | Float [0, 1] | +| `sig.hmac` | Optional | Base64-encoded HMAC-SHA256 | + +### Construction Code + +```python +from pmoves.tools.chit import build_cgp + +cgp = build_cgp( + anchors=U, # (K, D) anchor matrix from CHR + spectra=spectra, # (K, bins) from CHR + embeddings=embeddings, # (N, D) original embeddings + texts=texts, # N original text units + meta={ + "source": "text", + "units_mode": "sentences", + "K": 8, + "bins": 8, + "backend": "sentence-transformers/all-MiniLM-L6-v2" + } +) +``` + +See [CGP_ENCODING_REFERENCE.md](CGP_ENCODING_REFERENCE.md) for field-by-field construction details. + +--- + +## Stage 7: Security (Signing & Encryption) + +### HMAC Signing + +CGP packets are signed with HMAC-SHA256 for integrity and authenticity: + +```python +from pmoves.tools.chit_security import sign_cgp + +# Sign the packet +cgp_signed = sign_cgp(cgp, passphrase="shared-secret") + +# Verification +from pmoves.tools.chit_security import verify_cgp +is_valid = verify_cgp(cgp_signed, passphrase="shared-secret") +``` + +**Signing process:** + +1. Deep-copy the payload +2. Remove any existing `sig` block +3. Canonicalize: `json.dumps(payload, sort_keys=True, separators=(",", ":"))` +4. Compute: `HMAC-SHA256(canonical_json, passphrase)` +5. Attach `sig` block with `alg`, `kid` (first 16 chars of SHA256(passphrase)), and base64 `hmac` + +### Anchor Encryption + +For confidential content, anchor vectors can be encrypted with AES-GCM: + +```python +from pmoves.tools.chit_security import encrypt_anchors + +cgp_protected = encrypt_anchors(cgp_signed, passphrase="shared-secret") +# Each constellation's `anchor` field is replaced with `anchor_enc`: +# { +# "alg": "AES-GCM", +# "iv": "base64-iv", +# "salt": "base64-salt", +# "ct": "base64-ciphertext" +# } +``` + +With encrypted anchors, the CGP's geometric structure (spectra, positions) remains visible for routing and filtering, but the actual semantic content (anchor directions) is protected. + +--- + +## Stage 8: Transport (GEOMETRY BUS) + +CGP packets travel over the NATS-based GEOMETRY BUS. + +### Primary NATS Subjects + +| Subject | Publisher | Subscriber | Payload | +|---------|-----------|-----------|---------| +| `tokenism.cgp.ready.v1` | DeepResearch, SupaSerch, ToKenism | Hi-RAG v2, shape-store | Full CGP packet | +| `geometry.cgp.v1` | Hi-RAG v2, Gateway | Consumers | CGP transport | +| `geometry.event.v1` | Any | Persistence layer | Raw geometry persistence | +| `tokenism.attribution.recorded.v1` | ToKenism | Audit trail | Per-action attribution | +| `tokenism.cgp.weekly.v1` | ToKenism | Analytics | Weekly CGP export | +| `geometry.swarm.meta.v1` | EvoSwarm | Parameter consumers | Swarm optimization state | +| `tokenism.swarm.population.v1` | ToKenism | EvoSwarm | Attribution fairness metrics | + +### Publishing + +```python +import nats +import json + +async def publish_cgp(cgp): + nc = nats.NATS() + await nc.connect("nats://nats:pmoves@nats:4222") + await nc.publish( + "tokenism.cgp.ready.v1", + json.dumps(cgp).encode() + ) + await nc.close() +``` + +### JetStream Persistence + +CGP packets published to JetStream-backed subjects are durably stored: + +```bash +# Create stream for CGP persistence +nats stream add GEOMETRY_CGPS \ + --subjects "geometry.cgp.v1,tokenism.cgp.ready.v1" \ + --storage file \ + --retention limits \ + --max-msgs 100000 +``` + +--- + +## Stage 9: Decoding & Reconstruction + +### Three Decoding Modes + +#### Mode 1: Exact (Lossless) + +When `points[].text` or `points[].text_b64` is present, content is recovered directly: + +```python +decoded_text = base64.b64decode(point["text_b64"]).decode("utf-8") +# or simply: decoded_text = point["text"] +``` + +#### Mode 2: Geometry-Only (Lossy/Retrieval) + +When only geometric information is available (anchor, spectrum, radial_minmax), content is **reconstructed** from a shared corpus: + +``` +For each constellation: + 1. Project all corpus embeddings onto the anchor vector + 2. Bin projections into the same histogram structure as the spectrum + 3. Select corpus items whose projection distribution matches the target spectrum + 4. Rank by KL divergence between empirical and target distributions +``` + +This is the **Universal Codebook Property**: when encoder and decoder share the same embedding model and corpus, meaning can be reconstructed purely from geometry without transmitting raw tokens. + +#### Mode 3: Multi-Modal + +For image/audio/video CGPs, use the appropriate embedding model: + +```python +from pmoves.tools.chit import decode_images + +results = decode_images( + "cgp_clip.json", + image_dir="./images", + model_name="clip-ViT-B-32" +) +``` + +### Calibration Metrics + +After decoding, measure reconstruction quality: + +```python +from pmoves.tools.chit.chit_decoder import compute_metrics + +metrics = compute_metrics( + cgp=cgp, + decoded=results, + corpus_texts=texts, + corpus_vecs=vecs +) +``` + +| Metric | Good | Description | +|--------|------|-------------| +| KL Divergence | < 0.5 | Spectrum distribution match | +| JS Divergence | < 0.3 | Symmetric distribution distance | +| Wasserstein-1D | < 0.2 | Earth-mover distance | +| Coverage | > 0.8 | Fraction of content recovered | + +See [CALIBRATION_GUIDE.md](CALIBRATION_GUIDE.md) for detailed calibration procedures. + +--- + +## Stage 10: Swarm Optimization Feedback + +The pipeline is not one-shot — EvoSwarm continuously optimizes the encoding parameters. + +### Feedback Loop + +``` +CGP Telemetry → EvoSwarm Controller → Parameter Evolution → Updated Config + ↑ ↓ + └─────── New CGPs with better parameters ←──────────────────┘ +``` + +### What Gets Optimized + +| Parameter | Genome Field | Effect | +|-----------|-------------|--------| +| K (constellations) | `cg_builder.K` | Cluster granularity | +| bins (spectrum) | `cg_builder.bins` | Spectral resolution | +| tau (temperature) | `cg_builder.tau` | Assignment hardness | +| beta (regularization) | `cg_builder.beta` | Anchor stability | +| Decoder mode | `decoder.mode` | Reconstruction strategy | +| HRM threshold | `decoder.hrm_halt_thresh` | Refinement stopping | + +### Fitness Evaluation + +``` +fitness = weighted_sum( + reconstruction_quality, // KL divergence of decoded output + compression_ratio, // CGP size vs original content size + attribution_fairness, // Gini coefficient of weights + energy_efficiency // GPU watts * decode time +) +``` + +### Parameter Pack Distribution + +Optimized parameters are published as **parameter packs** via NATS: + +```json +{ + "pack_id": "pack-12345", + "namespace": "default", + "status": "active", + "best_fitness": 0.94, + "parameters": { + "cg_builder": { "K": 8, "bins": 8, "tau": 0.1, "beta": 0.01 }, + "decoder": { "mode": "swarm", "hrm_halt_thresh": 0.95 } + } +} +``` + +Published on `geometry.swarm.meta.v1` for all encoding services to consume. + +--- + +## Complete Data Flow Diagram + +``` + PMOVES.AI CHIT Mathematical Pipeline + ====================================== + + ┌─────────┐ ┌───────────┐ ┌─────────┐ ┌───────────┐ + │ Text │ │ Image │ │ Audio │ │ Video │ + │ Content │ │ Content │ │ Content │ │ Content │ + └────┬────┘ └─────┬─────┘ └────┬────┘ └─────┬─────┘ + │ │ │ │ + ▼ ▼ ▼ ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ STAGE 1: EMBEDDING │ + │ MiniLM-L6-v2 (text) | CLIP ViT-B-32 (image) | Whisper │ + │ Output: (N, D) normalized vectors │ + └──────────────────────────┬───────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ STAGE 2: CONSTELLATION HARVEST (CHR) │ + │ │ + │ Alternating Minimization: │ + │ E-step: assignments p[i][k] via softmax │ + │ M-step: anchors U[k] via weighted mean + normalize │ + │ │ + │ Output: K anchors, N×K assignments, K×bins spectra │ + └──────────────────────────┬───────────────────────────────────┘ + │ + ┌──────────┼──────────┐ + ▼ ▼ ▼ + ┌──────────────┐ ┌────────┐ ┌──────────────┐ + │ STAGE 3 │ │STAGE 4 │ │ STAGE 5 │ + │ Dirichlet │ │Poincare│ │ Zeta Filter │ + │ Weights │ │ Disk │ │ (gamma_k) │ + │ │ │ │ │ │ + │ alpha → w_i │ │(x,y,r) │ │ spectrum → │ + │ + Merkle │ │encoding│ │ spectrum_zeta│ + └──────┬───────┘ └───┬────┘ └──────┬───────┘ + │ │ │ + └─────────────┼──────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ STAGE 6: CGP PACKET CONSTRUCTION │ + │ │ + │ { spec: "chit.cgp.v1.0", meta: {...}, │ + │ super_nodes: [{ constellations: [{ │ + │ anchor, spectrum, spectrum_zeta, points: [{ │ + │ proj, conf, weight, merkle_proof, text │ + │ }] }] }] } │ + └──────────────────────────┬───────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ STAGE 7: SECURITY │ + │ sign_cgp() → HMAC-SHA256 sig block │ + │ encrypt_anchors() → AES-GCM anchor_enc (optional) │ + └──────────────────────────┬───────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ STAGE 8: GEOMETRY BUS (NATS) │ + │ tokenism.cgp.ready.v1 | geometry.cgp.v1 │ + │ JetStream-persisted for durable delivery │ + └────────────┬─────────────────────────────┬──────────────────┘ + │ │ + ▼ ▼ + ┌────────────────────┐ ┌────────────────────────┐ + │ STAGE 9: DECODE │ │ STAGE 10: EVOSWARM │ + │ │ │ FEEDBACK LOOP │ + │ Exact / Geometry / │ │ │ + │ Multi-modal │ │ Fitness evaluation → │ + │ │ │ Parameter evolution → │ + │ KL < 0.5 target │ │ Updated parameter pack │ + └────────────────────┘ └────────────────────────┘ +``` + +--- + +## Cross-References + +### Protocol Specifications +- [CGP v1.0 Specification](CGP_v1.0_SPECIFICATION.md) — Wire format definition +- [CGP Encoding Reference](CGP_ENCODING_REFERENCE.md) — Field-by-field construction guide +- [Calibration Guide](CALIBRATION_GUIDE.md) — KL/JS divergence calibration procedures + +### Mathematical Foundations +- [Integrating Math into PMOVES.AI](Integrating%20Math%20into%20PMOVES.AI.md) — Conceptual pillars +- [Three-Body Doctrine](THREE_BODY_DOCTRINE.md) — Philosophical framework +- [Constellation-Harvest-Regularization](Constellation-Harvest-Regularization/) — CHR deep dive + +### Implementation +- [TypeScript Modules](../../PMOVES-ToKenism-Multi/integrations/contracts/chit/) — 7 TS modules +- [CHIT Tools Catalog](../CHIT_TOOLS_CATALOG.md) — Python tooling reference +- [Mathematical UI Specification](Mathematical_UI_Design_Specification.md) — Visualization spec + +### Transport & Operations +- [GEOMETRY BUS Integration](GEOMETRY_BUS_INTEGRATION.md) — NATS integration guide +- [GEOMETRY NATS Subjects](../../.claude/context/geometry-nats-subjects.md) — Subject catalog +- [EvoSwarm Operations](../EVOSWARM_OPERATIONS_GUIDE.md) — Parameter optimization ops + +### Security +- [CHIT User Guide](CHIT_USER_GUIDE.md) — End-user signing/encryption guide +- [Security Patterns](../../.claude/context/security-patterns.md) — Cross-cutting security + +--- + +*This document is a living artifact tracked by [CHIT_CHANGE_TRACKER.md](../CHIT_CHANGE_TRACKER.md).*