Code

Add coverage deviation as a metric#417

Code

Merged

nikosbosse merged 56 commits intorework-add_coverage()epiforecasts/scoringutils:rework-add_coverage()from add-coverage-deviationepiforecasts/scoringutils:add-coverage-deviationCopy head branch name to clipboardNov 15, 2023

+1,869-1,777

Commits

Commits on Nov 10, 2023

• Add new function, interval_coverage_deviation_quantile() - I know the name is terrible

  nikosbosse

  committed
  31271d3

• Create a new helper function to compute ranges of central prediction intervals from quantiles

  nikosbosse

  committed
  9c9f286

• Update metrics_quantile

  nikosbosse

  committed
  b883dd8

• Small update to metrics_quantile + fixing a typo

  nikosbosse

  committed
  7b62919

• update ae_median_quantile() and documentation associated to wis() and other functions with the same arguments

  nikosbosse

  committed
  c674764

• add input checks to ae_median_sample() and se_mean_sample()

  nikosbosse

  committed
  27d4a78

• Update documentation for metrics_quantile and remove # no lint from other data documentation (seems it was ignored anyway)

  nikosbosse

  committed
  e2618cd

Commits on Nov 11, 2023

• Correct error in ae_median_quantile, update data file and documentation

  nikosbosse

  committed
  2ef9a52

• Correct an issue with pit() where this still relies on the computation of coverage_deviation per quantile as a metric

  nikosbosse

  committed
  7e635eb

• commit previously forgotten change in metrics_quantile

  nikosbosse

  committed
  77fcde4

Commits on Nov 12, 2023

• Fix issues with ae_median_sample(), ae_median_quantile() and se_mean_sample()

  nikosbosse

  committed
  295eb51

• Change "interval_score" to "wis" as a metric name. Comment out code for now that cannot be run anymore.

  nikosbosse

  committed
  88fcdce

• Replace score.scoringutils_quantile() with new function

  nikosbosse

  committed
  337ea1f

• update available_metrics() to include wis and coverage_ values (this is temporary)

  nikosbosse

  committed
  7e6dbe8

• Update vignette to update / comment out code that cannot be run anymore

  nikosbosse

  committed
  08dab31

• Fix test for ae_median_sample()

  nikosbosse

  committed
  f4d5d43

• Update tests

  nikosbosse

  committed
  9607615

• comment out / delete failing tests related to ranges in the output of score()

  nikosbosse

  committed
  c214ade

• Delete old version of score.score_quantile()

  nikosbosse

  committed
  71d6d65

• improve the way that data.frames are split in score() to deal with differing quantiles

  nikosbosse

  committed
  596dcee

• Add test file for binary metrics and input checks

  nikosbosse

  committed
  6989f5d

• Remove comment (to be turned into an issue)

  nikosbosse

  committed
  376de95

• small fix in score.scoringutils_quantile to avoid a warning

  nikosbosse

  committed
  b13f00a

• move tests around so they better correspond to file names used for actual functions

  nikosbosse

  committed
  983c030

• fix small test issues

  nikosbosse

  committed
  c89b1f5

Commits on Nov 13, 2023

• Add input checks for quantile-based forecasts

  nikosbosse

  committed
  cd0d154

• remove old test for sharpness (test has been moved to test-metrics-sample.R)

  nikosbosse

  committed
  40807ad

• move piece of code around within the metrics-quantile.R file (since metric has a many-to-one relationship)

  nikosbosse

  committed
  fa3a7ab

• Add more tests for quantile metrics

  nikosbosse

  committed
  0d42acc

• Rework existing add_coverage() function to work with raw forecasts

  nikosbosse

  committed
  32357f1

• Update News

  nikosbosse

  committed
  1be9055

• Update get_protetcted_columns() with coverage columns

  nikosbosse

  committed
  8719dbe

• Update quantile_to_interval.data.frame() to work with NA values

  nikosbosse

  committed
  1252e13

• Don't remove NA values in add_coverage() anymore

  nikosbosse

  committed
  3af9fff

• Update add_coverage to store an attribute metric_names with the name of the coverage columns

  nikosbosse

  committed
  02c4b1c

• Update tests and code snippets to get stuff working again that was previously failing due to add_coverage()

  nikosbosse

  committed
  b41d655

• Automatic readme update

  actions-user

  committed
  79fc533

Commits on Nov 14, 2023

• horrible, but working version of a refactoring of score.scoringutils_quantile

  nikosbosse

  committed
  f7faff2

• More elegant version, currently failing for sample-based forecasts because we can't pass the forecast_unit arg to apply_metrics

  nikosbosse

  committed
  cb9bcbe

• Fix score.scoringutils_sample by using matrices

  nikosbosse

  committed
  102370d

• move the expression into apply_metrics()

  nikosbosse

  committed
  004d2b8

• Update metrics_quantile (improved error handling + getting rid of run_safely()

  nikosbosse

  committed
  7eb7c96

• Add global variable to fix failing test

  nikosbosse

  committed
  6c213fc

• Create function to ensure an object is a data.table

  nikosbosse

  committed
  9705994

• Simplify set_forcast_unit and ensure that it operates on a data.table

  nikosbosse

  committed
  afdd3ac

• Make message in check_columns_present nicer

  nikosbosse

  committed
  218c01c

• Reintroduce run_safely() into metrics_quantile - the alternative wasn't working and everything else feels more complicated.

  nikosbosse

  committed
  21d887a

• correct check_columns_present()

  nikosbosse

  committed
  ebe3869

• Update set_forecast_unit() and add tests

  nikosbosse

  committed
  ba9adbf

Commits on Nov 15, 2023

• Merge pull request #419 from epiforecasts/rework_ae_median

  Show description for be1f53c
  nikosbosse

  authored
  be1f53c

• Merge pull request #421 from epiforecasts/rework-score.scoringutils_quantile()2

  Show description for c6c2dfc
  nikosbosse

  authored
  c6c2dfc

• Merge pull request #422 from epiforecasts/expand-tests

  Show description for 3928002
  nikosbosse

  authored
  3928002

• Merge pull request #423 from epiforecasts/expand-tests2

  Show description for b45ec5a
  nikosbosse

  authored
  b45ec5a

• Merge pull request #426 from epiforecasts/reword-add_coverage2

  Show description for e3fcec3
  nikosbosse

  authored
  e3fcec3

• Merge pull request #430 from epiforecasts/create-apply-metrics

  Show description for cc72c53
  nikosbosse

  authored
  cc72c53

• Merge pull request #437 from epiforecasts/fix-set_forecast_unit

  Show description for ad411da
  nikosbosse

  authored
  ad411da