SIMD-0061: Fair congestion control with intra-block exponential local base fee

proposals/0061-fair-congestion-control.md

@@ -0,0 +1,258 @@
+---
+simd: '0061'
+title: Fair congestion control with intra-block exponential local base fee
+authors:
+  - Ryo Onodera (Solana Labs)
+category: Standard/Meta
+type: Core
+status: Draft
+created: 2023-07-05
+feature: (fill in with feature tracking issues once accepted)
+---
+
+## Summary
+
+This feature is a fair congestion-control mechanism in the form of an extension
+to local fee markets while leaving their locality of transaction fee dynamics
+intact.
+
+To that end, it introduces a exponentially-scaled dynamic base fees to
+individual local fee markets. It also attains very short feedback loop of
+per-tx frequency to maintain full efficacy of Solana's peculiar execution
+model compared to other blockchains: multi-threaded and low latency.
+
+This is realized by means of some incentive tweaks to combat against the
+obvious base fee manipulation with such short intervals.
+
+## Motivation
+
+(still in half-finished...)
+
+- Write lock cu limit is bad (bot can lock out at the very first of block for
+  the entire duration of whole blocktime (400ms)
+- Increased Defi activities around any volatile financial markets could starve
+  payment transactions for extended time
+- Inter-block and linear Voluntary fee escalation with vanilla fee market
+  auction can't guarantee the scheduling deadline of casual payment txes (which
+  needs 99.99% sub-second confirmation at very minimum).
+
+## Overview of the Design
+
+This proposal tries to localize congestions by means of increasing minimum
+required `cu_price`s for each write-locked addresses (= `base_cu_price`), which
+is newly introduced by this proposal. This increase will be calculated
+exponentially, measured by the CU consumed by each addresses at the moment.
+This means a transaction must cost the sum of `requested_cu * base_cu_price`
+for all of its write-locked addresses at least. This results in selectively
+pricing out crowded subset of transactions waiting for block inclusion, while
+allowing other transactions to be processed for block inclusion.
+
+In this way, under-capitalized usage demand like spl-token transfers among
+wallets will still find its way for _timely_ block inclusion, accomplishing
+this mechanism's advertised _fairness_ in the sense of blockspace allocation
+and transaction fee isolation for each independent local fee markets even at
+the time of financial market's volatility, which entices well-capitalized usage
+demand in DeFi activities on chain. 
+
+This rate-limiting gets enforced only when the cluster deemed to be congested,
+meaning no idling block space is wasted when not congested. Also, those
+exponentially-increased `base_cu_price`s will be decreased exponentially
+likewise in terms of consumed CUs in a block (= `block_cu`), as long as its
+address-specific CUs (= `address_cu`) remain to be unchanged due to the
+temporal stalemate of relevant transaction processing. Collectively, each
+addresses get equal amount of opportunity to execute transactions in
+round-robin fashion while contributing to the decrease of other
+`base_cu_price`s, assuming no exponential priority-fee premium is paid among
+users to interrupt the orderly processing with such a high economical
+justification.
+
+On top of the direct appreciation of aforementioned fairness, this proposal
+also obsoletes both the existing block-wide CU limit and the account-write CU
+limit to overcome their inherent unfairness and problems. Also, no global
+`base_cu_price` is introduced for simplicity, relying on natural block-wide
+market-rate ceiling from individual active `base_cu_prices`s.
+
+Also, this proposal was conceived with the intent of reinforcement of Solana's
+multi-threaded, real-time, and low-latency transaction execution. Towards that
+high-achieving objective, rather drastic technical and economical changes are
+needed for the introduction of this consensus-level congestion control as
+described below in detail.
+
+## Detailed Design
+
+### Introductory example
+
+Due to this proposal's inherent complexities, the following very-simplified
+example is presented at first before diving into each design topic:
+
+Assume all transactions request 100k CU and always succeed after consuming 100k
+CU. Also assume the cluster is always congested.
+
+Available transactions for block inclusion (= `bufferd transactions`):
+
+- `Tx1_AB@9` (read: transaction numbered as `1`, which write-locks both address `A`
+  and address `B` with `cu_price == 9`)
+- `Tx2_A@8`
+- `Tx3_C@8`
+
+(1) 
+
+#### update freq
+
+why block interval is bad?
+  this proposal's primary objective is for congestion control, not like EIP 1559's price discovery
+  so non-interactiveness is desired for little or no room of active auctioning
+  so, quick and acute is desired much like tcp's rwin.
+
+### Incentive alignment
+
+
+
+
+
+
+### `base_cu_price` Calculation and block reward adjustment
+
+processing model is transactions are linearized by the order of appearance in ledger entries.
+
+### `reserved_fee` calculation and its accruing
+
+
+### Congestion Detection
+
+thread count or runnable/scheduled tx queue len?
+
+
+### Nonconflicting transaction group
+
+
+
+
+
+
+
+
+
+
+(i jotted this down in 10min before going to bed! pardon for being so random
+writings...)
+
+to *determiniscally* define active thread count (`TC_a`), additionally record
+transaction termination events into poh stream.
+
+also, derive stake-weighted average transaction execution thread count
+(`TC_stake_weighted`).
+
+so, full is defined as the duration when  `TC_a == TC_stake_weighted` (this is
+updated at ~10ms intervals).
+
+when not full, maximize throughput of each of any single threaded transaction
+executions. note that, this mode exponentially cools down any hot addresses if
+any.
+
+when full, effectively pause any txes touching the hot state by exponentially
+increasing the local base fees. so casual txes can be executed.
+
+so, leaders are incentivised to manipulate in this naive form.
+
+so, split priority fee into two parts: (1) collected, (2) accrued for the next
+tx's base fee payment. The portion of (1) is calculated as if tx's cu *
+`TC_stake_weighted`. (i.e. as if validator stuffed spam txes to capture the
+exessive part (2) of prirotiy fee)
+
+in this way, there's no meaning to spam blocks by leaders. at the same time,
+it's still incentivied to pack txes according to p.f. desceding order, because
+leaders and clients alike are want to increse of their single threaded tps)
+
+also requested fee is basis for fee cals, block fullness calc, not the actual
+cu.
+- to prevent bad behavior, rebate 50% of (requested CU - actual CU)?
+  - so that leaders want txes success (want to increase actual CU) => usually
+    execute in the order
+  - specified by user
+  - so that users want txes fail fast (want to decrease actual CU)
+  - 25% is burntd and 25% is collected to leaders
+    - so, avoid too much requested cu.
+
+finally, when substantial blocks are full for extended duration, the global base-fee
+will naturally starts ceiling up. That's unavoidable no matter what.
+
+priority fee isn't collected at all.
+banking can implement this congenstion mechanism without forced consensus rules if disired for experiment
+
+### Example
+
+`TC_stake_weighted == 10` 
+
+100 kcu
+
+tx1a -- tx2a -- tx3a
+
+tx1a:
+  cu 100kcu
+  base fee: 
+
+tx2a:
+  cu 200kcu
+
+tx3a:
+  cu 300kcu
+
+why leaders are incentized for picking more prioritized txes even if they only receive fixed base fees?
+  predictable auction mechanism and ceiling base fee as much as possible
+  compound reserve from firstly-executed txes?
+
+
+## Impact
+
+How will the implemented proposal impacts dapp developers, validators, and core contributors?
+
+## Security Considerations
+
+What security implications/considerations come with implementing this feature?
+Are there any implementation-specific guidance or pitfalls?
+
+## Drawbacks *(Optional)*
+
+Why should we not do this?
+
+## Backwards Compatibility *(Optional)*
+
+Does the feature introduce any breaking changes? All incompatibilities and
+consequences should be listed.
+
+## Alternatives Considered
+
+Related proposals:
+
+(TODO: add any relation of this to them)
+
+dynamic base fees
+https://github.com/solana-foundation/solana-improvement-documents/pull/4
+
+program rebatable account write fees:
+https://github.com/solana-foundation/solana-improvement-documents/pull/16
+
+asynchronous program execution:
+https://github.com/solana-foundation/solana-improvement-documents/pull/45
+
+increase prioritization fee:
+https://github.com/solana-foundation/solana-improvement-documents/pull/50
+
+bankless
+https://github.com/solana-foundation/solana-improvement-documents/pull/5
+
+## New Terminology
+
+Is there any new terminology introduced with this proposal?
+
+casual tx:
+fairness:
+block fullness in terms of number of actively-execution threads
+dark/filler tx: 
+tx base fee:
+address base fee:
+reserve <=> reward
+requested <=> required
+buffered
+

proposals/0061-fair-congestion-control/simulation/.gitignore

@@ -0,0 +1 @@
+/target

proposals/0061-fair-congestion-control/simulation/Cargo.toml

@@ -0,0 +1,4 @@
+[package]
+name = "fair-congestion-control-simulation"
+version = "0.1.0"
+edition = "2021"