Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create summary table of all GP benchmarks #5

Open
47 tasks
luiarthur opened this issue May 14, 2020 · 0 comments
Open
47 tasks

Create summary table of all GP benchmarks #5

luiarthur opened this issue May 14, 2020 · 0 comments
Milestone

Comments

@luiarthur
Copy link
Owner

luiarthur commented May 14, 2020

The GP models benchmark for a given sample size can be summarized in the table below, with the cells being compile times (if applicable) and execution times. Timings for forward passes and gradient computations of the models will also be computed if high-level API is available and exposed. The appropriate accompanying data visualizations (line plots, surface plots) will be included to make the results more interpretable. Links will be included to MWEs.

  • Make a dynamic table which changes N. (e.g. N=50, 100, 200, 400, 800).
  • Make interactive line plots where
    • x-axis is N, and y-axis is execution time or compile time
    • be able to select which implementations / PPL to show on the plot
  • Demo how GPs can be integrated & used as a building block within a probabilistic model.
Model (PPL), sample size = N Collapsed Gibbs SMC Particle Gibbs HMC/NUTS (SB) ADVI (SB)
GP (Turing)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
GP (STAN)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
GP (Nimble)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
GP (Pyro)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
GP (TFP) NA NA NA
  • TODO
  • TODO
LVGP (Turing)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
LVGP (STAN)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
LVGP (Nimble)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
LVGP (Pyro)
  • TODO
  • TODO
  • TODO
  • TODO
  • TODO
LVGP (TFP) NA NA NA
  • TODO
  • TODO

Notes:

  • Prefer doing the experiments by column as opposed to by row in this table. This way, if we are run out of time, we have benchmarks across PPLs, for a few implementations; and not benchmarks across a few implementations for one or two PPLs.
  • LVGP refers to latent variable GP. GP refer to vanilla (full-rank) Gaussian process.
@luiarthur luiarthur added this to the Third Month milestone May 14, 2020
@luiarthur luiarthur modified the milestones: Third Month, Second Month Jul 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant