Common Misconception in Clinical Prediction Model Research

doi:10.3969/j.issn.1673－5765.2023.07.004

Abstract

Abstract: Taking the inappropriate practices in existing research pointed out by a leading expert in clinical prediction models as examples, this paper introduced common misunderstandings in clinical prediction model research, explained why specific practices will jeopardize the model quality, and how to avoid repeating the same mistakes in future studies. The feasibility of using ChatGPT to provide methodological guidance is discussed by evaluating the answers given by ChatGPT.

Key words: Clinical prediction model; Methodological quality; Risk of bias

摘要： 本文结合临床预测模型领域专家指出的现有研究中的不恰当做法，介绍了临床预测模型研究中的常见误区，解析了为什么特定的做法会对模型质量产生不良影响，以及如何避免重蹈覆辙。同时，通过评价ChatGPT给出的答案，探讨ChatGPT用于提供方法学指导的可行性。

关键词: 临床预测模型; 方法学质量; 偏倚风险

WANG Haoyue, WANG Junfeng. Common Misconception in Clinical Prediction Model Research [J]. Chinese Journal of Stroke, 2023, 18(07): 758-769.

王昊玥, 王俊峰. 临床预测模型研究中的常见误区[J]. 中国卒中杂志, 2023, 18(07): 758-769.

References

[1] WYNANTS L，VAN CALSTER B，COLLINS G S，et al. Prediction models for diagnosis and prognosis of COVID-19：systematic review and critical appraisal[J/OL]. BMJ，2020，369[2023-01-15]. https://doi.org/10.1136/bmj.m1328.
[2] VAN SMEDEN M. The definitive guide to COVID-19 prognosis modeling success 2020[EB/OL]. [2023-03-28]. https://twitter.com/MaartenvSmeden/status/1262854476576194566.
[3] COLLINS G S，REITSMA J B，ALTMAN D G，et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis（TRIPOD）：the TRIPOD statement[J]. Ann Intern Med，2015，162（1）：55-63.
[4] MOONS K G，ALTMAN D G，REITSMA J B，et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis（TRIPOD）：explanation and elaboration[J/OL]. Ann Intern Med，2015，162（1）：W1-W73[2023-01-15]. https://doi.org/10.7326/M14-0698.
[5] WOLFF R F，MOONS K G，RILEY R D，et al. PROBAST：a tool to assess the risk of bias and applicability of prediction model studies[J]. Ann Intern Med，2019，170（1）：51-58.
[6] MOONS K G M，WOLFF R F，RILEY R D，et al. PROBAST：a tool to assess risk of bias and applicability of prediction model studies：explanation and elaboration[J/OL]. Ann Intern Med，2019，170（1）：W1-W33[2023-01-11]. https://doi.org/10.7326/M18-1377.
[7] ELDREDGE J D，WEAGEL E F，KROTH P J. Defining and identifying members of a research study population：CTSA-affiliated faculty members[J]. Hypothesis（Macon），2014，26（1）：5-11.
[8] VAN GELOVEN N，SWANSON S A，RAMSPEK C L，et al. Prediction meets causal inference：the role of treatment in clinical prediction models[J]. Eur J Epidemiol，2020，35（7）：619-630.
[9] PAJOUHESHNIA R，DAMEN J A A G，GROENWOLD R H H，et al. Treatment use in prognostic model research：a systematic review of cardiovascular prognostic studies[J/OL]. Diagn Progn Res，2017，1：15[2023-01-15]. https://doi.org/10.1186/s41512-017-0015-0.
[10] PROSEPE I，GROENWOLD R H，KNEVEL R，et al. The disconnect between development and intended use of clinical prediction models for COVID-19：a systematic review and real-world data illustration[J/OL]. Front Epidemiol，2022[2023-01-15]. https://doi.org/10.3389/fepid.2022.899589.
[11] WHITTLE R，PEAT G，BELCHER J，et al. Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported[J/OL]. J Clin Epidemiol，2018，102：38-49[2023-01-15]. https://doi.org/10.1016/j.jclinepi.2018.05.008.
[12] WHITTLE R，ROYLE K L，JORDAN K P，et al. Prognosis research ideally should measure time-varying predictors at their intended moment of use[J/OL]. Diagn Progn Res，2017，1：1[2023-01-15]. https://doi.org/10.1186/s41512-016-0006-6.
[13] OREŠKI G. An experimental comparison of classification algorithm performances for highly imbalanced datasets[C/OL]. Varaždin：25th Central European Conference on Information and Intelligent Systems，2014[2023-01-15]. https://xueshu.baidu.com/usercenter/paper/show?paperid=536441dc81b241ecc707d1c1fdf206d3&site=xueshu_se.
[14] JENI L A，COHN J F，DE LA TORRE F. Facing imbalanced data--recommendations for the use of performance metrics[C/OL]. Geneva：2013 Humaine Association Conference on Affective Computing and Intelligent Interaction，2013[2023-01-15]. https://xueshu.baidu.com/usercenter/paper/show?paperid=03a678e7b91668b70d17b4a502741f92&site=xueshu_se&hitarticle=1.
[15] CHOWDHURY M Z I，TURIN T C. Variable selection strategies and its importance in clinical prediction modelling[J/OL]. Fam Med Community Health，2020，8（1）：e000262[2023-01-15]. https://doi.org/10.1136/fmch-2019-000262.
[16] RAMSPEK C L，JAGER K J，DEKKER F W，et al. External validation of prognostic models：what，why，how，when and where?[J]. Clin Kidney J，2020，14（1）：49-58.
[17] RILEY R D，ENSOR J，SNELL K I E，et al. Calculating the sample size required for developing a clinical prediction model[J/OL]. BMJ，2020，368：m441[2023-01-15]. https://doi.org/10.1136/bmj.m441.
[18] PEDUZZI P，CONCATO J，KEMPER E，et al. A simulation study of the number of events per variable in logistic regression analysis[J]. J Clin Epidemiol，1996，49（12）：1373-1379.
[19] ARCHER L，SNELL K I E，ENSOR J，et al. Minimum sample size for external validation of a clinical prediction model with a continuous outcome[J]. Stat Med，2021，40（1）：133-146.
[20] RILEY R D，DEBRAY T P A，COLLINS G S，et al. Minimum sample size for external validation of a clinical prediction model with a binary outcome[J]. Stat Med，2021，40（19）：4230-4251.
[21] RILEY R D，COLLINS G S，ENSOR J，et al. Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome[J]. Stat Med，2022，41（7）：1280-1295.
[22] HAJIAN-TILAKI K. Receiver operating characteristic（ROC）curve analysis for medical diagnostic test evaluation[J]. Caspian J Intern Med，2013，4（2）：627-635.
[23] YING X. An overview of overfitting and its solutions[C/OL]. Bristol：IOP Publishing，2019[2023-01-15]. https://doi.org/10.1088/1742-6596/1168/2/022022.
[24] KOHAVI R，SOMMERFIELD D. Feature subset selection using the wrapper method：overfitting and dynamic search space topology[C/OL]. Palo Alto：AAAI Press，1995[2023-01-15]. https://xueshu.baidu.com/usercenter/paper/show?paperid=625841482e213ca48a1f651f0cbbf474&site=xueshu_se&hitarticle=1.
[25] SRIVASTAVA N，HINTON G，KRIZHEVSKY A，et al. Dropout：a simple way to prevent neural networks from overfitting[J]. J Mach Learn Res，2014，15（1）：1929-1958.
[26] DAEE P，PELTOLA T，VEHTARI A，et al. User modelling for avoiding overfitting in interactive knowledge elicitation for prediction[C/OL]. Tokyo：23rd International Conference on Intelligent User Interfaces，2018[2023-01-15]. https://doi.org/10.1145/3172944.3172989.
[27] FARRINGTON D P，LOEBER R. Some benefits of dichotomization in psychiatric and criminological research[J]. Crim Behav Ment Health，2010，10（2）：100-122.
[28] CHEN H N，COHEN P，CHEN S. Biased odds ratios from dichotomization of age[J]. Stat Med，2007，26（18）：3487-3497.
[29] AUSTIN P C，BRUNNER L J. Inflation of the type Ⅰ error rate when a continuous confounding variable is categorized in logistic regression analyses[J]. Stat Med，2004，23（7）：1159-1178.
[30] KWAK S K，KIM J H. Statistical data preparation：management of missing values and outliers[J]. Korean J Anesthesiol，2017，70（4）：407-411.
[31] DONDERS A R T，VAN DER HEIJDEN G J，STIJNEN T，et al. Review：a gentle introduction to imputation of missing values[J]. J Clin Epidemiol，2006，59（10）：1087-1091.
[32] KANG H. The prevention and handling of the missing data[J]. Korean J Anesthesiol，2013，64（5）：402-406.
[33] NATTINO G，PENNELL M L，LEMESHOW S. Assessing the goodness of fit of logistic regression models in large samples：a modification of the Hosmer-Lemeshow test[J]. Biometrics，2020，76（2）：549-560.
[34] PAUL P，PENNELL M L，LEMESHOW S. Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets[J]. Stat Med，2013，32（1）：67-80.
[35] EVERITT B，HOWELL D. Encyclopedia of Statistics in Behavioral Science[M]. New York：Wiley，2005.
[36] MOONS K G，ALTMAN D G，REITSMA J B，et al. New guideline for the reporting of studies developing，validating，or updating a multivariable clinical prediction model：the TRIPOD statement[J]. Adv Anat Pathol，2015，22（5）：303-305.
[37] HEUS P，DAMEN J A A G，PAJOUHESHNIA R，et al. Poor reporting of multivariable prediction model studies：towards a targeted implementation strategy of the TRIPOD statement[J/OL]. BMC Med，2018，16（1）：120[2023-01-15]. https://doi.org/10.1186/s12916-018-1099-2.
[38] LU J H，CALLAHAN A，PATEL B S，et al. Assessment of adherence to reporting guidelines by commonly used clinical prediction models from a single vendor：a systematic review[J/OL]. JAMA Netw Open，2022，5（8）：e2227779[2023-01-15]. https://doi.org/10.1001/jamanetworkopen.2022.27779.
[39] ALTMAN D G，VERGOUWE Y，ROYSTON P，et al. Prognosis and prognostic research：validating a prognostic model[J/OL]. BMJ，2009，338：b605[2023-01-15]. https://doi.org/10.1136/bmj.b605.
[40] CRADDOCK M，CROCKETT C，MCWILLIAM A，et al. Evaluation of prognostic and predictive models in the oncology clinic[J]. Clin Oncol，2022，34（2）：102-113.
[41] BLEEKER S E，MOLL H A，STEYERBERG E W，et al. External validation is necessary in prediction research：a clinical example[J]. J Clin Epidemiol，2003，56（9）：826-832.
[42] CAMPAGNER A，CAROBENE A，CABITZA F. External validation of machine learning models for COVID-19 detection based on complete blood count[J/OL]. Health Inf Sci Syst，2021，9（1）：37[2023-01-15]. https://doi.org/10.1007/s13755-021-00167-3.
[43] REPS J M，KIM C，WILLIAMS R D，et al. Can we trust the prediction model? Demonstrating the importance of external validation by investigating the COVID-19 vulnerability（C-19）index across an international network of observational healthcare datasets[J/OL]. medRxiv，2020[2023-01-15]. https://www.medrxiv.org/content/10.1101/2020.06.15.20130328v1.
[44] MARTENS F K，KERS J G，JANSSENS A C J. External validation is only needed when prediction models are worth it（Letter commenting on：J Clin Epidemiol. 2015; 68：25-34）[J/OL]. J Clin Epidemiol，2016，69：249-250[2023-01-20]. https://doi.org/10.1016/j.jclinepi.2015.01.022.
[45] STEYERBERG E W，BLEEKER S E，MOLL H A，et al. Internal and external validation of predictive models：a simulation study of bias and precision in small samples[J]. J Clin Epidemiol，2003，56（5）：441-447.
[46] KÖNIG I R，MALLEY J D，WEIMAR C，et al. Practical experiences on the necessity of external validation[J]. Stat Med，2007，26（30）：5499-5511.
[47] LOMBARDI Y，AZOYAN L，SZYCHOWIAK P，et al. External validation of prognostic scores for COVID-19：a multicenter cohort study of patients hospitalized in Greater Paris University Hospitals[J]. Intensive Care Med，2021，47（12）：1426-1439.