rpc: add percentile to getRecentPrioritizationFees

[fanatid]

Based on #30

docs about config in getRecentPrioritizationFees: https://docs.triton.one/chains/solana/improved-priority-fees-api

Add config with only one field percentile (valid value from 0 to 10_000) to getRecentPrioritizationFees. percentile allows to request custom min fee, for example:

{
  "method": "getRecentPrioritizationFees",
  "jsonrpc": "2.0",
  "params": [
    [
      "RNXnAJV1DeBt6Lytjz4wYzvS3d6bhsfidS5Np4ovwZz"
    ],
    {
      "percentile": 5000
    }
  ],
  "id": "1"
}

result:

{
  "jsonrpc": "2.0",
  "result": [
    {
      "slot": 348125,
      "prioritizationFee": 0
    },
    {
      "slot": 348126,
      "prioritizationFee": 1000
    },
    {
      "slot": 348127,
      "prioritizationFee": 500
    },
    {
      "slot": 348128,
      "prioritizationFee": 0
    },
    {
      "slot": 348129,
      "prioritizationFee": 1234
    }
  ],
  "id": 1
}

[CriesofCarrots]

@fanatid , sorry for the delay. Github didn't send a notification when you rebased 😕

@tao-stones , can you please also look at this, with attention to the new mem requirements and sort perf in the fee cache?

[CriesofCarrots]

I think we probably need a different word for this if it's actually based on 100 * 10^2, as percentiles are by definition based on 100.

At the least, we definitely need some docs for the field explaining the scale.

[CriesofCarrots]

Please fixup this unchecked math

[CriesofCarrots]

What's the benefit of all the duplicate methods? If we've written the apis right, I would think a get_transaction_fee(0) should yield the same result as get_min_transaction_fee(), so we can just update get_prioritization_fees to take percentile

[CriesofCarrots]

Suggested change

	let cache = match percentile {
	Some(percentile) => self
	.prioritization_fee_cache
	.get_prioritization_fees2(&pubkeys, percentile),
	None => self
	.prioritization_fee_cache
	.get_prioritization_fees(&pubkeys),
	};
	Ok(cache
	Ok(self
	.prioritization_fee_cache
	.get_prioritization_fees(&pubkeys, percentile.unwrap_or(0))

As per my comment on get_prioritization_fees2

[CriesofCarrots]

This is for metrics reporting... I don't think we want to stop on the writable_account_fees with the overall min_transaction_fee here.

[CriesofCarrots]

Please update the comment above (and any others that refer to pruning), since that no longer happens.

[tao-stones]

I can appreciate the motivation of this change, but uncomfortable of its implementation regarding additional memory consumption that could become attack vector.

[tao-stones]

Concern of memory footprint led to originally record only min and prune_irrelevant_writable_accounts when bank is OCed. Possible abusive scenario: 4000 txs/ block, 128 accounts/tx, then writable_account_fees would require additional sizeof(u64) * 4000 * 128 = 4MB per un-OCed block. I need more convincing that recording every transactions' and every accounts' prio fee will not be an attack vector.

Could histogram, or rolling EMA with variance (https://en.wikipedia.org/wiki/Moving_average#Exponentially_weighted_moving_variance_and_standard_deviation) be alternative? Perhaps user can query "ema plus n stddev"?

[tao-stones]

see comments about memory footprint

[tao-stones]

these sorting could be expensive, tho they are currently done in service thread, they should be OK?

[tao-stones]

this says the information stored is irrelevant: landing into block requires at least 2, implies write locking account_a would need >=2. so storing 2 for account_a has no additional information but wasting space.

[mschneider]

This can be solved in an external service like https://github.com/blockworks-foundation/lite-rpc, it's not really a good idea to add more functionality to the RPC API which is very bloated already. This feature could rather be specced out as a standard extension and then implemented for reference as a service that is compatible to the most popular validator clients.

[CriesofCarrots]

@fanatid , what's going on with this PR? Is it still something you'd like to move forward?

[fanatid]

I'd like to move forward if there is some chance that PR would be merged, just had no time to update :/

[fanatid]

separate service can be used for this kind of requests https://github.com/solana-stream-solutions/solfees

[mergify]

If this PR represents a change to the public RPC API:

1. Make sure it includes a complementary update to rpc-client/ (example)
2. Open a follow-up PR to update the JavaScript client @solana/web3.js (example)

Thank you for keeping the RPC clients in sync with the server API @fanatid.