Implement consistent naming and language for talking about scoring rules

[nikosbosse]

Update

We took the following decisions on what needs to be implemented

• we like "scoring rules" as a general term and will follow the language outlined in this post.
• we would call the default rules_point, rules_binary etc.
• argument for score() is rules
• argument for pairwise_comparison() is score

Things that need to be done:

• rename function arguments
• rename function apply_metrics to apply_rules
• update function documentation
• update Readme, Vignettes and NEWS file, adding an explanation of how we talk about things
• Update manuscript

---

Discussion:

We need a consistent naming scheme for

• talking about functions that compute scores
• the names of arguments related to those scoring functions
• the default functions used within score()

The two contenders were

• "scoring rules"
• "metrics"

Recently discussed this with @sbfnk.

• "Scoring rules" seemed a bit more precise
• "metrics" is shorter and in particular anything related to naming variables or anything doesn't have an underscore. Problem with metric is that it can be both the function and the output (i.e. computing a metric vs. the metric computes an output).

Proposal 1

"metric" would denote

• any function that computes a score / metric and that could be passed to score()
• sometimes maybe also the name of that function, e.g. "the absolute error is a scoring rule"

We would use sentences such as "you can pass a metric to score(), or "you can call a metric directly".

The proposal doesn't really have a good term to talk about the output of a "metric" (i.e., a scoring function). Calling the output of a "metric" also "metric" would maybe be a bit confusing.

The argument within score() would be called metrics

The argument within add_pairwise_comparison() that denotes what the pairwise comparison should be computed for would be called metric.

Default metrics could be the output of a function called default_metrics("point") or metrics_point()

Proposal 2

"scoring rule" would denote

• any function that computes a score / metric and that could be passed to score()
• sometimes maybe also the name of that function, e.g. "the absolute error is a scoring rule" or "crps" is a proper scoring rule

"a score"

• would denote the output of a "scoring rule"
• maybe sometimes also the the name of the scoring rule, like "The CRPS is a proper score" - but we should likely avoid these.

We would use sentences such as "you can pass a list of scoring rules to score(), or "you can call a scoring rule directly". And "scores are computed by scoring rules".

The argument within score() would be called scoring_rules

The argument within add_pairwise_comparison() that denotes what the pairwise comparison should be computed for would be called

• scoring_rule
• score?
• or maybe just for? ("What score should pairwise comparisons be computed for?")

Default scoring rules could be the output of a function called

• default_scores("point") or
• default_rules("point") or
• rules_point()

Alternative proposals

• could talk of "metrics" as the output of "metric functions"
• could do "scoring functions" instead of "scoring rules"

[nikosbosse]

Decisions:

• we like scoring rules
• we would call the default rules_point
• argument for score() isrules
• argument for pairwise_comparison() is score

[sbfnk]

argument for pairwise_comparison() is score

Should this be rule as suggested in #401? Otherwise there would be scores and score

[nikosbosse]

Phew... Probably rules or score_names? We have score_names in get_score_names() and also currently the attribute for the output from score() is called score_names.

[nikosbosse]

I'm going to close this in favour of #610 to limit the number of open issues that talk about the same thing...