CHIP-0013: Tighten Plot Filter Rules

[jmhands]

can be done independently or with the filter reduction described here #53

[danieljperry]

This CHIP is now a Draft. It will disallow plots from passing the plot filter for more than one out of any four consecutive signage points. Feel free to leave your reviews in this PR.

[maxjay]

Forgive my lack of understanding but would this mean there is a possibility of winning plots to effectively be ignored?

[danieljperry]

Forgive my lack of understanding but would this mean there is a possibility of winning plots to effectively be ignored?

You are correct that this CHIP would cause some plots that currently would produce winning proofs to no longer produce winning proofs. They would be ignored.

Keep in mind that currently, 511 out of every 512 plots are already ignored at each signage point. Some of those 511 plots would inevitably produce winning proofs if they were not ignored. The rationale for this CHIP is the same: every farmer will be "hurt", but they will all be hurt equally.

This proposal is quite powerful -- when an honest plot passes the filter, it has a 1 in 512 chance of passing the filter at the next signage point. But a grinded plot has a 512/512 (100%) chance of passing the second signage point. In this scenario, honest plots would sometimes be excluded (and all of them would be excluded equally) but grinded plots would always be excluded.

[SlowestTimelord]

This proposal is quite powerful -- when an honest plot passes the filter, it has a 1 in 512 chance of passing the filter at the next signage point. But a grinded plot has a 512/512 (100%) chance of passing the second signage point. In this scenario, honest plots would sometimes be excluded (and all of them would be excluded equally) but grinded plots would always be excluded.

Should be noted that although quite unlikely today, it does get more and more likely as the plot filter reduces if CHIP-12 gets implemented.

[madMAx43v3r]

That's a pretty good idea, much better than my current approach in MMX of doing 64 recursive hashes for the plot filter (to make filter grinding more expensive).

However a small mistake in the assumptions, this change will make 3 / plot_filter proofs invalid, not 1 / plot_filter^2 or 1 / plot_filter^3. So for a 256 filter it's 1.17 %, for 128 it's 2.34 % and for 32 it's 9.37 %.

Also, it will reduce effective netspace by some percentage at every plot filter change.

If you ask me, this change + filter reduction to 256 is enough to make grinding not profitable. HDD / SSD storage cost + power draw will go down over time as well, not just GPU compute cost.

[danieljperry]

Thanks for pointing this out, Max. The math in this CHIP presumes that a plot will be excluded if it passes each of the previous three filters, but it's actually excluded if it passes any of these filters. I will get the numbers updated.

[danieljperry]

This CHIP is in Review. Please make your final reviews soon.

[SlowestTimelord]

Thanks for adding the additional tables and specific heights as well as how it interacts with CHIP-12. I have no concerns with the CHIP itself but question about this:

Note that if the filter is reduced to be smaller than 32, this CHIP may need to be removed from the codebase. The reason for this is because the percent of plots excluded would continue to grow, thereby decreasing the effective netspace. In the most extreme scenario where the filter is removed entirely, this CHIP will also need to be removed -- it won't be possible to examine the previous filters when they don't exist.

Although it decreases effective netspace, it doesn't decrease actual security since everyone is still impacted equally, right? In fact removing the CHIP could decrease security by re-enabling the grinding attack simulated space multiplier.

[Voodoo-Chia]

Ok, let me see if I get this (no I don't, but still)

So if a plot grinder can go under 9.375 seconds for phase 1, they can make a single "plot" that will pass the filter 3 times in a row, so each spoofed plot gets 3 chances to win. Therefor, each grinded-plot spoofs 3*[current_filter] number of K32 plots.
Whereas 18,75 seconds gets you 1*[current_filter] number of K32 plots spoofed.

However grinding a plot that can pass the filter 3 times in a row, requires additional compute besides the creation time reduction. time due to the need to make the ID? required to pass each filter.

If above two are correct, then doesn't this becomes a question of how the compute scales for creating a single phase 1 (+extra hashes) in 9.375 sec vs 3 phase in 18.75 sec each? or even in general the compute difference of 1 fast vs 3 slower?

In other words: I have 3 magic GPU's that are each capable of creating a phase 1 in 18.75 seconds.
Will these three together be able to create a phase 1 in 9.375 seconds that passes all three challenges? Or do I maybe need 2 of them or 4?

[SlowestTimelord]

So if a plot grinder can go under 9.375 seconds for phase 1, they can make a single "plot" that will pass the filter 3 times in a row, so each spoofed plot gets 3 chances to win. Therefor, each grinded-plot spoofs 3*[current_filter] number of K32 plots. Whereas 18,75 seconds gets you 1*[current_filter] number of K32 plots spoofed.

Exactly, though with up to 18.75 seconds you'd get more like 2*[current_filter] plots spoofed.

However grinding a plot that can pass the filter 3 times in a row, requires additional compute besides the creation time reduction. time due to the need to make the ID? required to pass each filter.

Determining a Plot ID is the first step before starting to plot. Normally this is just generated at random but here the attacker would grind through a bunch of Plot IDs to find one that will pass the current & previous 1-2 signage points. The compute required to do this Plot ID grinding is negligible and would take milliseconds to do.

[madMAx43v3r]

The compute required to do this Plot ID grinding is negligible and would take milliseconds to do.

For the 3x gain it's not that easy, you need a pre-computed table of plot ids, like 10 * 512^3, which is around 80 GB of data to keep in RAM.

Generating a plot ID is quite a bit of compute due to BLS, hence you need this "rainbow" table for the 3x gain.

And then you still need to compute 512^3 sha256 on average per plot. Which requires to transfer ~4 GB of data to a GPU on average, unless you have a bunch of GPUs holding 40 GB in VRAM together (or a single A100 / H100).

It's a fun project for sure.

[madMAx43v3r]

Is it not 3 / filter_size ? Almost the same though.

[danieljperry]

That is a fairly close estimate, but not exact. To visualize the difference, imagine a filter_size of 2. In that case, 3 / filter_size would give a 150% chance of a plot being excluded, which isn't possible.

The equation in the CHIP calculates the probability of a plot not being excluded three times in a row, and then subtracts that from 1 in order to get the correct probability.

[Voodoo-Chia]

It's a fun project for sure.
😂

So what do you think of the difference in compute required between a x3 grind vs 3 single grinds?
Because if they are anywhere near equal in total compute requirement, that would mean this Chip isn't very useful.

[madMAx43v3r]

Because if they are anywhere near equal in total compute requirement, that would mean this Chip isn't very useful.

The compute to find a x3 plot ID is still negligible compared to the phase 1 compute itself. So you are getting a nearly 3x gain, because every plot you generate has a 3x higher chance to win a block.

This CHIP is for real, I'm doing the same in MMX now.

[Voodoo-Chia]

Cool, thanks for that clarification

[SlowestTimelord]

Generating a plot ID is quite a bit of compute due to BLS, hence you need this "rainbow" table for the 3x gain.

And then you still need to compute 512^3 sha256 on average per plot. Which requires to transfer ~4 GB of data to a GPU on average, unless you have a bunch of GPUs holding 40 GB in VRAM together (or a single A100 / H100).

It's a fun project for sure.

Ah true, I once again have underestimated exponentials.

[danieljperry]

Thanks for adding the additional tables and specific heights as well as how it interacts with CHIP-12. I have no concerns with the CHIP itself but question about this:

Note that if the filter is reduced to be smaller than 32, this CHIP may need to be removed from the codebase. The reason for this is because the percent of plots excluded would continue to grow, thereby decreasing the effective netspace. In the most extreme scenario where the filter is removed entirely, this CHIP will also need to be removed -- it won't be possible to examine the previous filters when they don't exist.

Although it decreases effective netspace, it doesn't decrease actual security since everyone is still impacted equally, right? In fact removing the CHIP could decrease security by re-enabling the grinding attack simulated space multiplier.

Correct, it's just a semantic decrease, but not an actual decrease in security.

[danieljperry]

This CHIP is now in Last Call. If no further changes are required in the next two weeks (roughly), it will be moved to Final.

[danieljperry]

We have decided to move this CHIP back to Review for further testing.

[danieljperry]

This CHIP's activation height has been updated to 4 603 536, which is expected to occur in December 2023.

[SlowestTimelord]

As mentioned in the AMA today, the reported estimated netspace is expected to go down ~1% as a result of this CHIP (and more in future plot filter reductions) due to slightly fewer eligible plots passing filter. Has there been consideration given to updating the netspace estimation formula to adjust for this decrease?

[danieljperry]

As mentioned in the AMA today, the reported estimated netspace is expected to go down ~1% as a result of this CHIP (and more in future plot filter reductions) due to slightly fewer eligible plots passing filter. Has there been consideration given to updating the netspace estimation formula to adjust for this decrease?

Good point, this is indeed something we should consider.

[danieljperry]

This CHIP's activation height has been updated to 5 496 000, which is expected to occur in June 2024.

[neurosis69]

What happens if I have multiple plots sharing the same plot_id, but with different k-sizes?
Will all plots being disallowed from passing the filter on consecutive challenges?

[neurosis69]

let me rephrase my still unanswered questions :)

would it be possible to make this CHIP-13 only effective for plots where the k=32?
or introduce a variable to limit this CHIP up to a specified k value only?
When assuming that plot grinding is currently not viable for plots where k > 32

[danieljperry]

13 on

It's a good thought, but I don't think this is possible because the plot filter is based on the plot ID. The filter has no knowledge of the k-size of the plots.

[danieljperry]

We need to withdraw this CHIP from consideration for now.

After performing a considerable amount of testing, we have decided that the tradeoffs of the potential for a farmer to affect future blocks by withholding the current block are not worth the gains in making it more costly to grind. The primary reason for this is that the signage points are not canonical to the chain.

However, if grinding ever becomes an issue, we could still add this CHIP's implementation to a new CHIP. We also will consider making the signage points canonical to the chain in order to mitigate against the aforementioned tradeoff.

[neurosis69]

13 on

It's a good thought, but I don't think this is possible because the plot filter is based on the plot ID. The filter has no knowledge of the k-size of the plots.

but if I've filled a disk only with plots of the same plot id but using a different k-value to ensure getting different proofs, then this CHIP would have affected all the plots, even the ones which probably weren't a potential target for grinding (with k > 32).
Just wanted to add this background why I've asked, as this CHIP may be re-opened somewhere in the future.

[hoffmang9]

I do not believe differing k value plots can have the same plot ID. I know that any plots of the same k value with the same plot ID are simply wasted space.

[neurosis69]

I'm reluctant to disagree, but different k-values create different proofs even for the same Plot ID.
If I'm not mistaken, then a k32 creates 2^32 32 bit numbers, where a k33 creates 2^33 33 bit numbers and so on.

I have 3 x 16 TB HDDs plotted like this:

(venv) chia@harvester:~$ ls -l /plots/disk3/
total 15550479368
drwx------ 2 root root         16384 Aug 29 08:58 lost+found
-rw-r--r-- 1 chia chia  108861755679 Sep  7 14:12 plot-k32-2023-09-07-13-26-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot
-rw-r--r-- 1 chia chia  224249604930 Sep  7 13:32 plot-k33-2023-09-07-10-14-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot
-rw-r--r-- 1 chia chia  461545059442 Sep  7 18:20 plot-k34-2023-09-07-14-06-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot
-rw-r--r-- 1 chia chia  949262654553 Sep  8 00:30 plot-k35-2023-09-07-17-48-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot
-rw-r--r-- 1 chia chia 1950732636450 Sep  5 16:10 plot-k36-2023-09-05-01-35-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot
-rw-r--r-- 1 chia chia 4006604336733 Sep  9 05:47 plot-k37-2023-09-07-23-24-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot
-rw-r--r-- 1 chia chia 8222433289664 Sep 12 07:18 plot-k38-2023-09-09-22-25-9464a65cf7e5f0eaa037ff15abeef987e2bbfb9befe63e29b1f8cc90062f34c0.plot

You can review the farm here:
https://spacefarmers.io/farmers/7961789d1412a6e4c18c7d2694578e35fcfd8bdec0a61ecf64ddafce895f04b7/harvesters

It is the harvester called big-k-plots, where you can clearly see by clicking on the dots, which plot with which k-size created a partial. I use this method currently, to put the HDDs into standby, saving about 80% power (using 1W on standby). With the current plot filter the HDD is only woken up about 18 times a day.

[madMAx43v3r]

I do not believe differing k value plots can have the same plot ID. I know that any plots of the same k value with the same plot ID are simply wasted space.

The K size is not hashed into the plot ID, so it's possible. Different K size with same ID still have different proofs, so it's not wasted space.

[madMAx43v3r]

then this CHIP would have affected all the plots, even the ones which probably weren't a potential target for grinding (with k > 32).

It doesn't matter, what you have is effectively one large plot, like a K39, split into multiple files. So this CHIP would not affect you at all.