-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathfit_ppmc.Rd
137 lines (124 loc) · 6.18 KB
/
fit_ppmc.Rd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/ppmc.R
\name{fit_ppmc}
\alias{fit_ppmc}
\title{Posterior predictive model checks for assessing model fit}
\usage{
fit_ppmc(
model,
ndraws = NULL,
probs = c(0.025, 0.975),
return_draws = 0,
model_fit = c("raw_score"),
item_fit = c("conditional_prob", "odds_ratio", "pvalue"),
force = FALSE
)
}
\arguments{
\item{model}{A \link{measrfit} object.}
\item{ndraws}{The number of posterior draws to base the checks on. Must be
less than or equal to the total number of posterior draws retained in the
estimated model. If \code{NULL} (the default) the total number from the
estimated model is used.}
\item{probs}{The percentiles to be computed by the \code{\link[stats:quantile]{stats::quantile()}}
function for summarizing the posterior distributions of the specified fit
statistics.}
\item{return_draws}{Proportion of posterior draws for each specified fit
statistic to be returned. This does not affect the calculation of the
posterior predictive checks, but can be useful for visualizing the fit
statistics. For example, if \code{ndraws = 500}, \code{return_draws = 0.2}, and
\code{model_fit = "raw_score"}, then the raw score chi-square will be computed
500 times (once for each draw) and 100 of those values (0.2 * 500) will be
returned. If \code{0} (the default), only summaries of the posterior are
returned (no individual samples).}
\item{model_fit}{The posterior predictive model checks to compute for an
evaluation of model-level fit. If \code{NULL}, no model-level checks are
computed. See details.}
\item{item_fit}{The posterior predictive model checks to compute for an
evaluation of item-level fit. If \code{NULL}, no item-level checks are computed.
Multiple checks can be provided in order to calculate more than one check
simultaneously (e.g., \code{item_fit = c("conditional_prob", "odds_ratio")}).
See details.}
\item{force}{If all requested \acronym{PPMC}s have already been added to the
model object using \code{\link[=add_fit]{add_fit()}}, should they be recalculated. Default is
\code{FALSE}.}
}
\value{
A list with two elements, "model_fit" and "item_fit". If either
\code{model_fit = NULL} or \code{item_fit = NULL} in the function call, this will be
a one-element list, with the null criteria excluded. Each list element, is
itself a list with one element for each specified \acronym{PPMC} containing
a \link[tibble:tibble-package]{tibble}. For example if
\code{item_fit = c("conditional_prob", "odds_ratio")}, the "item_fit" element
will be a list of length two, where each element is a tibble containing the
results of the \acronym{PPMC}. All tibbles follow the same general
structure:
\itemize{
\item \verb{obs_\{ppmc\}}: The value of the relevant statistic in the observed
data.
\item \code{ppmc_mean}: The mean of the \code{ndraws} posterior samples calculated
for the given statistic.
\item Quantile columns: 1 column for each value of \code{probs}, providing the
corresponding quantiles of the \code{ndraws} posterior samples calculated for
the given statistic.
\item \code{samples}: A list column, where each element contains a vector of
length \code{(ndraws * return_draws)}, representing samples from the
posterior distribution of the calculated statistic. This column is
excluded if \code{return_draws = 0}.
\item \code{ppp}: The posterior predictive p-value. This is the proportion of
posterior samples for calculated statistic that are greater than the
observed value. Values very close to 0 or 1 indicate incompatibility
between the fitted model and the observed data.
}
}
\description{
For models estimated with \code{method = "mcmc"}, use the posterior distributions
to compute expected distributions for fit statistics and compare to values
in the observed data.
}
\details{
Posterior predictive model checks (PPMCs) use the posterior distribution of
an estimated model to compute different statistics. This creates an expected
distribution of the given statistic, \emph{if our estimated parameters are
correct}. We then compute the statistic in our observed data and compare the
observed value to the expected distribution. Observed values that fall
outside of the expected distributions indicate incompatibility between the
estimated model and the observed data.
We currently support PPMCs at the model and item level. At the model level,
we calculate the expected raw score distribution (\code{model_fit = "raw_score"}),
as described by Thompson (2019) and Park et al. (2015).
At the item level, we can calculate the conditional probability that a
respondent in each class provides a correct response (\code{item_fit = "conditional_prob"}) as described by Thompson (2019) and Sinharay & Almond
(2007) or the overall proportion correct for an item (\code{item_fit = "pvalue"}),
as described by Thompson (2019). We can also calculate the odds ratio for
each pair of items (\code{item_fit = "odds_ratio"}) as described by Park et al.
(2015) and Sinharay et al. (2006).
}
\examples{
\dontshow{if (measr_examples()) (if (getRversion() >= "3.4") withAutoprint else force)(\{ # examplesIf}
mdm_dina <- measr_dcm(
data = mdm_data, missing = NA, qmatrix = mdm_qmatrix,
resp_id = "respondent", item_id = "item", type = "dina",
method = "mcmc", seed = 63277, backend = "rstan",
iter = 700, warmup = 500, chains = 2, refresh = 0
)
fit_ppmc(mdm_dina, model_fit = "raw_score", item_fit = NULL)
\dontshow{\}) # examplesIf}
}
\references{
Park, J. Y., Johnson, M. S., Lee, Y-S. (2015). Posterior
predictive model checks for cognitive diagnostic models. \emph{International
Journal of Quantitative Research in Education, 2}(3-4), 244-264.
\doi{10.1504/IJQRE.2015.071738}
Sinharay, S., & Almond, R. G. (2007). Assessing fit of cognitive
diagnostic models. \emph{Educational and Psychological Measurement, 67}(2),
239-257. \doi{10.1177/0013164406292025}
Sinharay, S., Johnson, M. S., & Stern, H. S. (2006). Posterior
predictive assessment of item response theory models. \emph{Applied
Psychological Measurement, 30}(4), 298-321.
\doi{10.1177/0146621605285517}
Thompson, W. J. (2019). \emph{Bayesian psychometrics for diagnostic
assessments: A proof of concept} (Research Report No. 19-01). University
of Kansas; Accessible Teaching, Learning, and Assessment Systems.
\doi{10.35542/osf.io/jzqs8}
}