Skip to content

network: stateful vote compression#6351

Merged
cce merged 18 commits intoalgorand:masterfrom
cce:vpack-dynamic
Oct 16, 2025
Merged

network: stateful vote compression#6351
cce merged 18 commits intoalgorand:masterfrom
cce:vpack-dynamic

Conversation

@cce
Copy link
Copy Markdown
Contributor

@cce cce commented May 30, 2025

Summary

This adds to #6276 a StatefulEncoder and StatefulDecoder, that compress votes using HPACK-style references to values from previous votes. It provides better than 50% compression given different table sizes.

Notes on memory vs bandwidth tradeoff and effectiveness of stateful vote compression are here: #6467

Test Plan

  • Unit tests in vpack package
  • Ensure full test coverage in vpack package

@codecov
Copy link
Copy Markdown

codecov bot commented May 30, 2025

Codecov Report

Attention: Patch coverage is 98.43444% with 8 lines in your changes missing coverage. Please review.

Project coverage is 50.65%. Comparing base (18990e0) to head (571d707).
Report is 61 commits behind head on master.

Files with missing lines Patch % Lines
network/vpack/dynamic_vpack.go 97.97% 6 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6351      +/-   ##
==========================================
- Coverage   51.82%   50.65%   -1.17%     
==========================================
  Files         652      656       +4     
  Lines       87425   110899   +23474     
==========================================
+ Hits        45307    56177   +10870     
- Misses      39249    51858   +12609     
+ Partials     2869     2864       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cce cce requested a review from Copilot July 12, 2025 05:13

This comment was marked as outdated.

algorandskiy
algorandskiy previously approved these changes Aug 6, 2025
@cce cce requested a review from Copilot August 6, 2025 18:41
@cce cce marked this pull request as ready for review August 6, 2025 18:41
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request introduces dynamic vote compression functionality using StatefulEncoder and StatefulDecoder components that implement HPACK-style references to values from previous votes, providing over 50% compression efficiency with different table sizes.

  • Adds StatefulEncoder/StatefulDecoder with LRU tables for caching commonly used values (addresses, public key/signature pairs)
  • Implements a proposal window using HPACK-style indexing for proposal bundle compression
  • Introduces delta encoding for round numbers and reference-based compression for repeated values

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
network/vpack/vpack.go Updates header comment and refactors varuint parsing logic into a shared helper function
network/vpack/msgp.go Adds msgpVaruintRemaining helper function and refactors readUintBytes to use it
network/vpack/dynamic_vpack.go Core implementation of StatefulEncoder/StatefulDecoder with compression logic
network/vpack/lru_table.go Implements 2-way set-associative LRU hash table for value caching
network/vpack/proposal_window.go Implements HPACK-style sliding window for proposal bundle references
network/vpack/dynamic_vpack_test.go Comprehensive tests for stateful encoding/decoding functionality
network/vpack/lru_table_test.go Tests for LRU table operations including eviction and consistency
network/vpack/proposal_window_test.go Tests for proposal window HPACK-style behavior

@gmalouf gmalouf requested review from gmalouf and jannotti August 8, 2025 19:02
gmalouf
gmalouf previously approved these changes Aug 15, 2025
@cce cce dismissed stale reviews from gmalouf and algorandskiy via a58a9c2 October 7, 2025 21:07
@cce cce merged commit 1625da7 into algorand:master Oct 16, 2025
39 checks passed
cce added a commit that referenced this pull request Nov 3, 2025
This follows #6351 by integrating it with the WebsocketNetwork implementation, using a new VP message tag.

Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: John Jannotti <jannotti@gmail.com>
@cce cce changed the title network: dynamic vote compression network: dynamic / stateful vote compression Nov 6, 2025
@cce cce changed the title network: dynamic / stateful vote compression network: stateful vote compression Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants