chore: pippenger int audit

barretenberg/cpp/CLAUDE.md

@@ -2,6 +2,14 @@ succint aztec-packages cheat sheet.
 
 THE PROJECT ROOT IS AT TWO LEVELS ABOVE THIS FOLDER. Typically, the repository is at ~/aztec-packages. all advice is from the root.
 
+# Git workflow for barretenberg
+
+**IMPORTANT**: When comparing branches or looking at diffs for barretenberg work, use `merge-train/barretenberg` as the base branch, NOT `master`. The master branch is often outdated for barretenberg development.
+
+Examples:
+- `git diff merge-train/barretenberg...HEAD` (not `git diff master...HEAD`)
+- `git log merge-train/barretenberg..HEAD` (not `git log master..HEAD`)
+
 Run ./bootstrap.sh at the top-level to be sure the repo fully builds.
 Bootstrap scripts can be called with relative paths e.g. ../barretenberg/bootstrap.sh
 

barretenberg/cpp/scripts/compare_branch_vs_baseline_remote.sh

@@ -16,7 +16,7 @@ PRESET=${3:-clang20}
 BUILD_DIR=${4:-build}
 HARDWARE_CONCURRENCY=${HARDWARE_CONCURRENCY:-16}
 
-BASELINE_BRANCH="master"
+BASELINE_BRANCH="${BASELINE_BRANCH:-merge-train/barretenberg}"
 BENCH_TOOLS_DIR="$BUILD_DIR/_deps/benchmark-src/tools"
 
 if [ ! -z "$(git status --untracked-files=no --porcelain)" ]; then

barretenberg/cpp/src/barretenberg/ecc/fields/field_declarations.hpp

@@ -189,13 +189,47 @@ template <class Params_> struct alignas(32) field {
         return { data[0], data[1], data[2], data[3] };
     }
 
+    /**
+     * @brief Extract a slice of bits from raw limbs (no Montgomery conversion)
+     * @details Returns bits [lo_bit, hi_bit) from the raw limb representation.
+     *          Useful for algorithms like Pippenger MSM that need to slice scalars
+     *          that have already been converted out of Montgomery form.
+     *
+     * @param lo_bit Starting bit position (inclusive)
+     * @param hi_bit Ending bit position (exclusive)
+     * @return uint32_t The extracted bit slice
+     */
+    [[nodiscard]] constexpr uint32_t get_bit_slice_raw(size_t lo_bit, size_t hi_bit) const noexcept
+    {
+        BB_ASSERT_DEBUG(lo_bit < hi_bit);
+        BB_ASSERT_DEBUG(hi_bit <= FIELD_BITS);
+
+        // Optimized: directly access limbs without creating temporary uint256_t objects (~4x faster)
+        size_t start_limb = lo_bit / LIMB_BITS;
+        size_t end_limb = hi_bit / LIMB_BITS;
+        size_t lo_slice_offset = lo_bit & (LIMB_BITS - 1);
+        size_t slice_size = hi_bit - lo_bit;
+        size_t lo_slice_bits = (LIMB_BITS - lo_slice_offset < slice_size) ? (LIMB_BITS - lo_slice_offset) : slice_size;
+        size_t hi_slice_bits = slice_size - lo_slice_bits;
+
+        uint64_t lo_slice = (data[start_limb] >> lo_slice_offset) & ((1ULL << lo_slice_bits) - 1);
+        uint64_t hi_slice = (start_limb != end_limb) ? (data[end_limb] & ((1ULL << hi_slice_bits) - 1)) : 0;
+
+        return static_cast<uint32_t>(lo_slice | (hi_slice << lo_slice_bits));
+    }
+
     constexpr field(const field& other) noexcept = default;
     constexpr field(field&& other) noexcept = default;
     constexpr field& operator=(const field& other) & noexcept = default;
     constexpr field& operator=(field&& other) & noexcept = default;
     constexpr ~field() noexcept = default;
     alignas(32) uint64_t data[4]; // NOLINT
 
+    // Limb size constants for field element representation
+    static constexpr size_t NUM_LIMBS = 4;
+    static constexpr size_t LIMB_BITS = 64;
+    static constexpr size_t FIELD_BITS = 256;
+
     static constexpr uint256_t modulus =
         uint256_t{ Params::modulus_0, Params::modulus_1, Params::modulus_2, Params::modulus_3 };
 #if defined(__SIZEOF_INT128__) && !defined(__wasm__)

barretenberg/cpp/src/barretenberg/ecc/scalar_multiplication/README.md

@@ -0,0 +1,178 @@
+# Pippenger Multi-Scalar Multiplication (MSM)
+
+## Overview
+
+The Pippenger algorithm computes multi-scalar multiplications:
+
+$$\text{MSM}(\vec{s}, \vec{P}) = \sum_{i=0}^{n-1} s_i \cdot P_i$$
+
+**Complexity**: $O(\frac{q}{c}(n + 2^c))$ group operations. With optimal $c \approx \frac{1}{2} \log_2 n$, this is roughly $O(n \cdot q / \log n)$, vs $O(n \cdot q)$ for naive scalar multiplication.
+
+## Algorithm
+
+### Step 1: Scalar Decomposition
+
+**Implementation**: `get_scalar_slice(scalar, round_index, bits_per_slice)`
+
+Each scalar $s_i$ is decomposed into $r$ slices of $c$ bits each, processed **MSB-first**:
+
+$$s_i = \sum_{j=0}^{r-1} s_i^{(j)} \cdot 2^{c(r-1-j)}$$
+
+- $c$ = bits per slice (from `get_optimal_log_num_buckets`, which brute-force searches for minimum cost)
+- $r = \lceil $ `NUM_BITS_IN_FIELD` $/ c \rceil$ = number of rounds
+- Round 0 extracts the most significant bits
+
+### Step 2: Bucket Accumulation
+
+For each round $j$, points are added into **buckets** based on their scalar slice. Bucket $k$ accumulates all points whose slice value equals $k$:
+
+$$B_k^{(j)} = \sum_{\{i : s_i^{(j)} = k\}} P_i$$
+
+**Two implementation paths:**
+
+- **Affine**: Sorts points by bucket and uses batched affine additions
+- **Jacobian**: Direct bucket accumulation in Jacobian coordinates
+
+### Step 3: Bucket Reduction
+
+**Implementation**: `accumulate_buckets(bucket_accumulators)`
+
+Computes weighted sum using a suffix sum (high to low):
+
+$$R^{(j)} = \sum_{k=1}^{2^c - 1} k \cdot B_k^{(j)} = \sum_{k=1}^{2^c - 1} \left( \sum_{m=k}^{2^c - 1} B_m^{(j)} \right)$$
+
+An offset generator is added and subtracted to avoid rare accumulator edge cases—a probabilistic mitigation that simplifies accumulation logic.
+
+### Step 4: Round Combination
+
+Combines all rounds using Horner's method (MSB-first):
+
+```cpp
+msm_accumulator = point_at_infinity
+for j = 0 to r-1:
+    repeat c doublings (or fewer for final round)
+    msm_accumulator += bucket_result[j]
+```
+
+## Algorithm Variants
+
+### Entry Points and Safety
+
+| Entry Point | Default | Safety |
+|-------------|---------|--------|
+| `msm()` | `handle_edge_cases=false` | ⚠️ **Unsafe** |
+| `pippenger()` | `handle_edge_cases=true` | ✓ Safe |
+| `pippenger_unsafe()` | `handle_edge_cases=false` | ⚠️ Unsafe |
+| `batch_multi_scalar_mul()` | `handle_edge_cases=true` | ✓ Safe |
+
+### Edge Cases
+
+Affine addition fails for **P = Q** (doubling), **P = −Q** (inverse), and **P = O** (identity). Jacobian coordinates handle these correctly at higher cost (~2-3× slower).
+
+⚠️ **Use `msm()` or `pippenger_unsafe()` only when points are guaranteed linearly independent** (e.g., SRS points). For user-controlled or potentially duplicate points, use `pippenger()`.
+
+### Affine Pippenger (`handle_edge_cases=false`)
+
+Uses affine coordinates with Montgomery's batch inversion trick: replaces $m$ inversions with **1 inversion + O(m) multiplications**, yielding ~2-3× speedup over Jacobian.
+
+### Jacobian Pippenger (`handle_edge_cases=true`)
+
+Uses Jacobian coordinates for bucket accumulators. Handles all edge cases correctly.
+
+## Tuning Constants
+
+| Constant | Value | Purpose |
+|----------|-------|---------|
+| `PIPPENGER_THRESHOLD` | 16 | Below this, use naive scalar multiplication |
+| `AFFINE_TRICK_THRESHOLD` | 128 | Below this, batch inversion overhead exceeds savings |
+| `MAX_SLICE_BITS` | 20 | Upper bound on bucket count exponent |
+| `BATCH_SIZE` | 2048 | Points per batch inversion (fits L2 cache) |
+| `RADIX_BITS` | 8 | Bits per radix sort pass |
+
+<details>
+<summary>Cost model constants and derivations</summary>
+
+| Constant | Value | Derivation |
+|----------|-------|------------|
+| `BUCKET_ACCUMULATION_COST` | 5 | 2 Jacobian adds/bucket × 2.5× cost ratio |
+| `AFFINE_TRICK_SAVINGS_PER_OP` | 5 | ~10 muls saved − ~3 muls for product tree |
+| `JACOBIAN_Z_NOT_ONE_PENALTY` | 5 | Extra field ops when Z ≠ 1 |
+| `INVERSION_TABLE_COST` | 14 | 4-bit lookup table for modular exp |
+
+**BATCH_SIZE=2048**: Each `AffineElement` is 64 bytes. 2048 points = 128 KB, fitting in L2 cache.
+
+**RADIX_BITS=8**: 256 radix buckets × 4 bytes = 1 KB counting array, fits in L1 cache.
+
+</details>
+
+## Implementation Notes
+
+### Zero Scalar Filtering
+
+`transform_scalar_and_get_nonzero_scalar_indices` filters out zero scalars before processing (since $0 \cdot P_i = \mathcal{O}$). Scalars are converted from Montgomery form in-place to avoid doubling memory usage.
+
+### Bucket Existence Tracking
+
+A `BitVector` bitmap tracks which buckets are populated, avoiding expensive full-array clears between rounds. Clearing the bitmap costs $O(2^c / 64)$ words vs $O(2^c)$ for the full bucket array.
+
+### Point Scheduling (Affine Variant Only)
+
+Entries are packed as `(point_index << 32) | bucket_index` and sorted via **in-place MSD radix sort**. Sorting groups points by bucket, enabling efficient batch processing. The sort also detects entries with `bucket_index == 0` during the final radix pass, allowing zero-bucket entries to be skipped without a separate scan.
+
+### Batched Affine Addition
+
+`batch_accumulate_points_into_buckets` processes sorted points iteratively:
+- Same-bucket pairs → queue for batch addition
+- Different buckets → cache in bucket or queue with existing accumulator
+- Uses branchless conditional moves to minimize pipeline stalls
+- Prefetches future points to hide memory latency
+- Recirculates results to maximize batch efficiency before writing to buckets
+
+<details>
+<summary>Batch accumulation case analysis</summary>
+
+| Condition | Action | Iterator Update |
+|-----------|--------|-----------------|
+| `bucket[i] == bucket[i+1]` | Queue both points for batch add | `point_it += 2` |
+| Different buckets, accumulator exists | Queue point + accumulator | `point_it += 1` |
+| Different buckets, no accumulator | Cache point into bucket | `point_it += 1` |
+
+After batch addition, results targeting the same bucket are paired again before writing to bucket accumulators, reducing random memory access by ~50%.
+
+</details>
+
+## Parallelization
+
+Uses **per-thread buffers** (bucket accumulators, scratch space) to eliminate contention.
+
+For `batch_multi_scalar_mul()`, work is distributed via `MSMWorkUnit` structures that can split a single MSM across multiple threads. Each thread computes partial results on point subsets, combined in a final reduction.
+
+<details>
+<summary>Per-call buffer sizes</summary>
+
+| Buffer | Size | Purpose |
+|--------|------|---------|
+| `BucketAccumulators` (affine) | $2^c × 64$ bytes | Affine bucket array + bitmap |
+| `JacobianBucketAccumulators` | $2^c × 96$ bytes | Jacobian bucket array + bitmap |
+| `AffineAdditionData` | ~400 KB | Scratch for batch inversion |
+| `point_schedule` | $n × 8$ bytes | Per-MSM point schedule |
+
+Buffers are allocated per-call for WASM compatibility. Memory scales with thread count during parallel execution.
+
+</details>
+
+## File Structure
+
+```
+scalar_multiplication/
+├── scalar_multiplication.hpp    # MSM class, data structures
+├── scalar_multiplication.cpp    # Core algorithm
+├── process_buckets.hpp/cpp      # Radix sort
+├── bitvector.hpp                # Bit vector for bucket tracking
+└── README.md                    # This file
+```
+
+## References
+
+1. Pippenger, N. (1976). "On the evaluation of powers and related problems"
+2. Bernstein, D.J. et al. "Faster batch forgery identification" (batch inversion)

barretenberg/cpp/src/barretenberg/ecc/scalar_multiplication/bitvector.hpp

@@ -54,6 +54,13 @@ class BitVector {
         std::memset(data_.data(), 0, data_.size() * sizeof(uint64_t));
     }
 
+    void resize(size_t num_bits)
+    {
+        num_bits_ = num_bits;
+        data_.resize((num_bits + 63) / 64);
+        clear();
+    }
+
     size_t size() const { return num_bits_; }
 
     // Optional: access raw pointer for performance