External Validation of the COVID-NoLab and COVID-SimpleLab Prognostic Tools

INTRODUCTION

Accurate, easy-to-use prognostic tools may help optimize management decisions for patients with COVID-19, potentially reducing inpatient burden by identifying patients at low risk for inpatient death. Most previous risk scores include laboratory tests or imaging studies that make them cumbersome and time-consuming to implement.^1-4 Simpler risk scores have been proposed^2-5 but have not been externally validated in North America. We therefore developed and internally validated the COVID-NoLab risk score (which uses only clinical findings) and the COVID-SimpleLab risk score (which uses clinical findings and simple laboratory measures) in a US population to predict mortality among inpatients with COVID-19.⁶ The objective of the current study was to externally validate these risk scores.

METHODS

We obtained a deidentified database of 637 adults with confirmed COVID-19 infection from the Biobanque Québécoise de la COVID-19 (https://www.bqc19.ca/), selecting patients who presented to 11 hospitals in Quebec, Canada between March 8, 2020 and February 22, 2021 (85% were admitted).⁷ Patients with missing data were excluded, leaving 588 patients for validation of the COVID-NoLab score and 479 patients for validation of the COVID-SimpleLab score. The risk scores and their components are summarized in Table 1.

Each patient in the Quebec cohort was categorized as having low, moderate, or high risk for mortality using the original cutoffs identified for the COVID-NoLab and COVID-SimpleLab tools. Classification accuracy was assessed by determining the mortality rate in each risk group. The area under the receiver operating characteristic curve (AUROC) was used as a measure of overall discrimination. We compared accuracy in patients admitted between March 8, 2020 and July 14, 2020 (early) and in patients admitted between July 15, 2020 and February 22, 2021 (late) to evaluate prediction stability over time. The Pearson χ² test was used to compare proportions, the Welch t test to compare means, and the Mann-Whitney U test to compare medians between groups. Analysis was performed using Stata version 17.0 (Stata Corp).

RESULTS

Patients in the Quebec cohort were significantly older than those in the original US derivation population, aged 66.0 years vs 60.4 years in the COVID-NoLab sample and aged 66.7 years vs 61.4 years in the COVID-SimpleLab sample. Mortality was also slightly lower overall in both the COVID-NoLab sample (10.5% vs 13.8%, P = .05) and in the COVID-SimpleLab sample (11.3% vs 14.3%, P = .95). There were no clinically important differences between groups with regard to respiratory and laboratory parameters (Supplemental Table 1).

Both risk scores validated well, with no clinically meaningful differences in the mortality rates for each risk group between the original derivation cohort and the Quebec cohort, other than a somewhat lower mortality in the high-risk group (Table 2). In the Quebec cohort, the COVID-NoLab model identified a low-risk group with 1 death in 95 patients, while the COVID-SimpleLab model identified a low-risk group with 0 deaths in 126 patients. The COVID-NoLab model had AUROC of 0.72 in the Quebec cohort, while the COVID-SimpleLab model had an AUROC of 0.73. Full data are shown in Supplemental Table 2.

There were no significant differences in classification accuracy between the early and late subcohorts (Supplemental Table 3). Among the 102 patients in the Quebec COVID-NoLab sample who were not admitted, there was only 1 death in a patient classified as moderate risk (Supplemental Table 4) and no deaths in the Quebec COVID-SimpleLab sample. We also developed and internally validated models for settings in which clinicians have access to only the white blood cell count or only the C-reactive protein level (available on request). Although both models were able to identify high-risk patients, in each case, the low-risk group in the validation data sets had an appreciably higher mortality rate than the corresponding group in the derivation data set (4.4% vs 0.0% for both models).

DISCUSSION

We have validated 2 previously reported COVID-19 risk scores that are simple and rapid to use, demonstrating generalizability across patient populations at different time points in the pandemic. The COVID-NoLab model does not require laboratory testing and patients in the low-risk group had only 1% mortality, while no deaths were seen in the low-risk group for the COVID-SimpleLab score. Use of these tools may help decrease unnecessary admissions during COVID-19 surges.

Another study evaluated the COVID-NoLab and COVID-SimpleLab risk scores in a French population of 14,343 inpatients and found AUROC values similar to those in the Quebec population (0.70 for COVID-NoLab and 0.71 for COVID-SimpleLab).⁸ For the COVID-NoLab score, mortality rates were 1.5% in the low-risk group, 14.6% in the moderate-risk group, and 31.1% in the high-risk group. For the COVID-SimpleLab score, the mortality rates were 2.4% in the low-risk group, 12.8% in the moderate-risk group, and 28.2% in the high-risk group.

Strengths of our study include provision of 2 models that use somewhat different data, enabling clinicians with different laboratory access to make accurate COVID-19 risk stratifications; also, these risk scores have now been successfully externally validated in 2 other countries and at different times in the pandemic. A limitation is that overall discrimination was not quite as good in the validation cohorts (0.70-0.73 vs 0.80-0.81 in the derivation cohort). Both risk scores, however, successfully identified clinically useful low-risk groups in the new cohort. It will also be important to continue to test the models over time to ensure their accuracy as new variants emerge. The risk scores have not been validated in an outpatient setting, so it is not known whether they predict clinical deterioration in outpatients with COVID-19. Among patients discharged home from the emergency department in the Quebec cohort, there was only 1 death among those classified as having moderate or high risk. The overall clinical impression of physicians therefore is an important factor, and as for any clinical prediction rule, these scores should be used only to supplement that judgement.

The COVID-NoLab and COVID-SimpleLab risk scores have been successfully validated in 2 new populations of hospitalized patients from different countries and at later times in the pandemic. Appropriate use in conjunction with clinical judgement may decrease unnecessary hospital admissions for low-risk patients. We have made the COVID-NoLab risk score available at https://ebell-projects.shinyapps.io/covid_nolab/ and the COVID-SimpleLab risk score at https://ebell-projects.shinyapps.io/COVID-SimpleLab/.