Hypothesis testing with multiple data splits and exchangeable p-values
Statistical tests are sometimes constructed with data splitting. When such tests are applied to data, the result can depend on the way the data is split, which is typically random. Therefore, on a dataset, the result of a test is random and not replicable. Further, such tests typically have low power because the full sample is not utlized.
R package MultiSplit properly aggregates the results from multiple data splits and reports the p-value of the aggregated statistic. The constructed test has level that asymptotically approaches the nominal level. Typically, by aggregating results from a sufficiently large number of data splits, the test becomes replicable and much more powerful. This package implements a generic method that handles any test that is constructed with "extra randomness", including random data splitting, resampling, imputation, etc.
The package can be installed from GitHub.
# install.packages("devtools")
# devtools::install_github("cran/kedd")
devtools::install_github("richardkwo/MultiSplit")In case of problem, first make sure dependency kedd is properly installed. It is removed from CRAN but can be found from CRAN archive or GitHub.
devtools::install_github("cran/kedd")For a quick start, check out the vignette in folder vignette/ and the documentation for main function test.multisplit.
The package supports parallelized subsampling with package future. The mode of parallelization is controlled by command plan() (e.g., plan(multisession)). See also here.
Folder scripts/ contains code for reproducing numerical experiments.
- v0.1.1
- Fixed malfunction of
kedddue to newifbehavior in R 4.2+
- Fixed malfunction of
- v0.1.0
- Intial release
Guo, F. Richard, and Rajen D. Shah. "Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values." arXiv:2301.02739 (2023).