Recommendations for algorithmic fairness assessments of predictive models in healthcare : evidence from large-scale empirical analyses
More options
Description
Creators/Contributors
- Author/Creator
- Pfohl, Stephen Robert, author.
- Contributor
- Shah, Nigam, degree supervisor.
- Goel, Sharad, 1977- degree committee member.
- Hernandez-Boussard, Tina degree committee member.
- Zou, James, degree committee member.
- Stanford University. Department of Biomedical Informatics.
Contents/Summary
- Summary
- The use of machine learning to develop predictive models that inform clinical decision making has the potential to introduce and exacerbate health inequity. A growing body of work has framed these issues as ones of algorithmic fairness, seeking to develop techniques to anticipate and proactively mitigate harms. The central aim of my work is to provide and justify practical recommendations for the development and evaluation of clinical predictive models in alignment with these principles. Using evidence derived from large-scale empirical studies, I demonstrate that, when it is assumed that the predicted outcome is not subject to differential measurement error across groups and threshold selection is unconstrained, approaches that aim to incorporate fairness considerations into the learning objective used for model development typically do not improve model performance or confer greater net benefit for any of the studied patient populations compared to standard learning paradigms. For evaluation in this setting, I advocate for the use of criteria that assess the calibration properties of predictive models across groups at clinically-relevant decision thresholds. To contextualize the interplay between measures of model performance, fairness, and benefit, I present a case study for models that estimate the ten-year risk of atherosclerotic cardiovascular disease to inform statin initiation. Finally, I caution that standard observational analyses of algorithmic fairness in healthcare lack the contextual grounding and causal awareness necessary to reason about the mechanisms that lead to health disparities, as well as about the potential for technical approaches to counteract those mechanisms, and argue for refocusing algorithmic fairness efforts in healthcare on participatory design, transparent model reporting, auditing, and reasoning about the impact of model-enabled interventions in context
Bibliographic information
- Publication date
- 2021
- Copyright date
- 2021
- Note
- Submitted to the Department of Biomedical Informatics
- Note
- Thesis Ph.D. Stanford University 2021
