[Feature request] p-values table

[nikohansen]

$p$-values are currently coarsely coded in the tables as the first digit of $\max(\lg(1/p))$, where only the best algorithm is annotated. When we post-process more than two algorithms, we compute all pairwise $p$-values and we could create a table from these which provides additional information and is reasonably informative on its own. As we have less than $k^2/2$ tests to report for $k$ algorithms, we have still $k^2/2$ cells where we could report some other relevant piece of information.

Proposal: The table should report $\pm\log_2(1/p)$ from the two-sided test without Bonferroni correction (which represents the number of coin flips ending up head - probability) and the sign indicates which algorithm is better. Reading out may be easier if the value is put in the lower left part of the matrix, because then we can always start picking an algorithm from the left-most column. The unused cells could give the relative geometric average runtime ratios, and the diagonal could give the runtime ratio w.r.t. the best as in the original tables.

Open decisions

• $p$-values are placed in the upper right or lower left?
• sort algorithms by runtime or alphabetically?
• rounding method for displaying $\log_2(1/p)$ (we don't present decimal places?)
• display also success numbers or rates?

Any other/better ideas what to display?

The feature should create a new html page with one $p$-value table for each function and each dimension.

[brockho]

I am in favour of such tables. The p-values for each algorithm pair will be handy: like that I won't have to run an additional postprocessing for two algorithms of interest :-)

The feature should create a new html page with one value table for each function and each dimension.

Don't we actually have one table per function/dimension/target pair?

[nikohansen]

Don't we actually have one table per function/dimension/target pair?

Not sure for the table per ... pair, but we should have (a few) different targets, which raises a page layout question. All targets in one cell?