Multi-level longitudinal learning curve regression models integrated with item difficulty metrics for deliberate practice of visual diagnosis: groundwork for adaptive learning

Abstract

Visual diagnosis of radiographs, histology and electrocardiograms lends itself to deliberate practice, facilitated by large online banks of cases. Which cases to supply to which learners in which order is still to be worked out, with there being considerable potential for adapting the learning. Advances in statistical modeling, based on an accumulating learning curve, offer methods for more effectively pairing learners with cases of known calibrations. Using demonstration radiograph and electrocardiogram datasets, the advantages of moving from traditional regression to multilevel methods for modeling growth in ability or performance are demonstrated, with a final step of integrating case-level item-response information based on diagnostic grouping. This produces more precise individual-level estimates that can eventually support learner adaptive case selection. The progressive increase in model sophistication is not simply statistical but rather brings the models into alignment with core learning principles including the importance of taking into account individual differences in baseline skill and learning rate as well as the differential interaction with cases of varying diagnosis and difficulty. The developed approach can thus give researchers and educators a better basis on which to anticipate learners’ pathways and individually adapt their future learning.

Appendices

Appendix 1

Variable Definition and Notation

See Table 4

Table 4 Notation table for variables, parameters and indexes

Appendix 2

Comments on Local Independence

Conditional independence is an important limiting assumption of both IRT and traditional linear models (De Boeck, 2004). Specifically, the errors in a regression model are assumed to be independent of each other. This may be more or less plausible depending on what is conditioned on in the model. For instance, once we include the covariate vector ${\mathbf{x}}$, the fixed effects for diagnosis (or item, or item order) this assumption applies to only the portion of the response not explained by these inputs, making this independence assumption more plausible because it allows responses to be more similar to each other within groups defined by the inputs (for instance among responses for items with the same diagnosis).

A critical advantage of multilevel modeling is the ability to explicitly account for non-independence of observations even beyond this. For instance the random effects associated with each person p in our models allow for (a specific type of) correlation across measurements for the same person beyond the fact that we predict that they will have a similar average value. They allow us to make a more flexible assumption of a correlation of measures within person.

Our models do not include this type of explicit correlation within items or even within diagnosis. However, in a learning context with longitudinal data, we expect the knowledge gained on early items will improve learners’ accuracy on subsequent items, especially of the same diagnosis. For our final model in Eq. 14 ("Discussion" section), we have assumed that responses of a single learner at a given time are independent of previous responses, conditioned on specific learner parameters (learning rate and base knowledge). In the data we used for this paper, items were reviewed by the learners in a random order and we measured the learning trajectory adjusted for the difficulty of the items within diagnosis. Through the inclusion of the diagnoses’ fixed effects, we ameliorate some of the problem in that we allow items with a similar diagnosis to share a common mean. Additional correlation may exist, and moreover, the time trajectory could induce further correlation.

Future models could consider other covariance structures to account for response dependence, for example by assuming the learning rate at future times will depend on the amount of learning up to that specific point in time. Another strategy would be to include random effects for diagnosis therefore allowing us to distinguish specific item effects within the different diagnosis groups. Our intent is not to map out all possibilities but rather to point out that the multi-level modeling context has the flexibility to address a limiting assumption of the traditional logistic model.

Appendix 3

Model Evaluation

This appendix evaluates and compares the models used to develop the nonlinear learning trajectories – the multilevel logistic models from Eqs. 6, 8 and 14. The primary goal of this comparison is to evaluate whether adjusting for the diagnosis helps explain the learning in greater detail and provides more accurate learner-level predictions.

We first evaluate relative model performance by comparing the information criteria Akaike Information Criteria and Bayesian Information Criteria (AIC and BIC) across the models. As we add informative predictors to our models, we expect these measures to improve (i.e. decrease). As Table 4 demonstrates, all three criteria do indeed favor the learning trajectory models that incorporate diagnosis information.

See Table 5

Table 5 Model diagnostics comparison

To further evaluate our models, we can perform simulations to generate new data corresponding to the assumptions implicit in the model. (Gelman and Hill 2007a) For each simulated dataset we can calculate relevant summaries of the data. We use two such statistics. The first two are the mean and standard deviation of the total number of correct responses at the population level (across learners—see Fig. 10, Elbow Dataset). We then compare the same statistics in our observed data to the distribution of statistics formed from our simulated datasets. This evaluation will help us understand whether our modeling assumptions are reasonable for the data we are trying to understand. Said another way, we can see whether it is reasonable to assume our observed data arose from a world meeting the assumptions of the model.

See Fig. 10.

Fig. 10

Elbow dataset. The test statistic used for this plot is the mean and standard deviation of the total number of correct responses by learner. The red line represents the true observed value in our training data; the blue line the simulated result. From left to right, we have the complete pooling model (no hierarchical structure on the parameters), the random intercepts and slopes with no diagnosis, and the random intercepts and slopes with the diagnosis terms. Top Panel: Mean; Bottom Panel Standard Deviation

We can see in Fig. 10a and b the results of our first two test statistics for the three models we compared on Table 2. These demonstrate that although the complete pooling approach closely reproduces the mean of the total correct responses it underestimates the standard deviation of this summary statistic by an important margin. Also note that, when we include random intercept and slopes, the additional variance terms help us correct this behavior and we not only are able to get a better fit to the standard deviation, but to the mean as well. Both multilevel models closely reproduce the first summary statistic and the second falls well within the predictive simulated distributions for both models. The datasets and code necessary to repeat these analyses is available at: https://github.com/IlanReinstein/ecg_paper