Add G-Pass@k Metric

[jnanliu]

This PR aims to support the G-Pass@k from paper.

G-Pass@k is a generalized version of Pass@k, measuring the ability of models to generate m correct solution in k attempts, where m is controled by the parameter thresholds. When the threshold is 0, G-Pass@k will discard to G-Pass@k. G-Pass@k can measure the potential and stability of models simultaneously.

$$ \text{G-Pass@}k_{\tau} = E_{\text{Questions}} \left[ \sum_{j = \lceil \tau \cdot k \rceil}^{c} \frac{\binom{c}{j} \cdot \binom{n - c}{k - j}}{\binom{n}{k}} \right] $$

$$ \text{mG-Pass@}k_{\tau} = 2\int_{0.5}^{1.0} \text{G-Pass@}k_{\tau} d \tau = \frac{2}{k} \sum_{i= \lceil 0.5 \cdot k \rceil + 1}^{k} \text{G-Pass@}k_{\frac{i}{k}} $$

[NathanHB]

hey ! Thanks for the PR :)
Do you plan to also add the math benchmark that comes with it ?

[jnanliu]

hey, I have added some tasks in tasks/default_tasks.py that support G-Pass@16 evaluation on AIME24/25 and MATH500 benchmarks, you can check it :)

[tonysy]

@NathanHB Hi, I would like to know if this PR can be merged now? or does it need more further modification?

[NathanHB]

Looks good ! Only a verry small nit, wait for the tests and good to merge

[NathanHB]

list instead of List

[jnanliu]

have fixed it

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[jnanliu]

@NathanHB Hi, I have fixed the nit, please allow the checks :)