Replicability, Reproducibility, and Agent-based Simulation of Interventions

R. Stanley Hum, Samantha Kleinberg

Research output: Contribution to journalArticle

Abstract

Secondary use of medical data and use of observational data for causal inference has been growing. Yet these data bring many challenges such as confounding due to unobserved variables and variation in medical processes across settings. Further, while methods exist to handle some of these problems, researchers lack ground truth to evaluate these methods. When a finding is not replicated across multiple sites, it is unknown whether this is a failure of an algorithm, a genuine difference between populations, or an artifact of structural differences between the sites. We show how agent-based simulation of medical interventions can be used to explore how bias, error, and variation across settings affect inference. Our approach enables users to model not only interventions and outcomes, but also the complex interaction between patients with different risks of mortality and providers with different observed and latent treatment effects. Ultimately we propose that such simulations can be used to better evaluate the behavior of new methods with known ground truth and better calculate sample size for EHR-based studies.

Original languageEnglish (US)
Pages (from-to)959-968
Number of pages10
JournalAMIA ... Annual Symposium proceedings. AMIA Symposium
Volume2017
StatePublished - Jan 1 2017

Fingerprint

Sample Size
Artifacts
Research Personnel
Mortality
Population
Therapeutics

Cite this

@article{b5faa8aaca354ae9896b15121441ef16,
title = "Replicability, Reproducibility, and Agent-based Simulation of Interventions",
abstract = "Secondary use of medical data and use of observational data for causal inference has been growing. Yet these data bring many challenges such as confounding due to unobserved variables and variation in medical processes across settings. Further, while methods exist to handle some of these problems, researchers lack ground truth to evaluate these methods. When a finding is not replicated across multiple sites, it is unknown whether this is a failure of an algorithm, a genuine difference between populations, or an artifact of structural differences between the sites. We show how agent-based simulation of medical interventions can be used to explore how bias, error, and variation across settings affect inference. Our approach enables users to model not only interventions and outcomes, but also the complex interaction between patients with different risks of mortality and providers with different observed and latent treatment effects. Ultimately we propose that such simulations can be used to better evaluate the behavior of new methods with known ground truth and better calculate sample size for EHR-based studies.",
author = "Hum, {R. Stanley} and Samantha Kleinberg",
year = "2017",
month = "1",
day = "1",
language = "English (US)",
volume = "2017",
pages = "959--968",
journal = "AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium",
issn = "1559-4076",
publisher = "American Medical Informatics Association",

}

Replicability, Reproducibility, and Agent-based Simulation of Interventions. / Hum, R. Stanley; Kleinberg, Samantha.

In: AMIA ... Annual Symposium proceedings. AMIA Symposium, Vol. 2017, 01.01.2017, p. 959-968.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Replicability, Reproducibility, and Agent-based Simulation of Interventions

AU - Hum, R. Stanley

AU - Kleinberg, Samantha

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Secondary use of medical data and use of observational data for causal inference has been growing. Yet these data bring many challenges such as confounding due to unobserved variables and variation in medical processes across settings. Further, while methods exist to handle some of these problems, researchers lack ground truth to evaluate these methods. When a finding is not replicated across multiple sites, it is unknown whether this is a failure of an algorithm, a genuine difference between populations, or an artifact of structural differences between the sites. We show how agent-based simulation of medical interventions can be used to explore how bias, error, and variation across settings affect inference. Our approach enables users to model not only interventions and outcomes, but also the complex interaction between patients with different risks of mortality and providers with different observed and latent treatment effects. Ultimately we propose that such simulations can be used to better evaluate the behavior of new methods with known ground truth and better calculate sample size for EHR-based studies.

AB - Secondary use of medical data and use of observational data for causal inference has been growing. Yet these data bring many challenges such as confounding due to unobserved variables and variation in medical processes across settings. Further, while methods exist to handle some of these problems, researchers lack ground truth to evaluate these methods. When a finding is not replicated across multiple sites, it is unknown whether this is a failure of an algorithm, a genuine difference between populations, or an artifact of structural differences between the sites. We show how agent-based simulation of medical interventions can be used to explore how bias, error, and variation across settings affect inference. Our approach enables users to model not only interventions and outcomes, but also the complex interaction between patients with different risks of mortality and providers with different observed and latent treatment effects. Ultimately we propose that such simulations can be used to better evaluate the behavior of new methods with known ground truth and better calculate sample size for EHR-based studies.

UR - http://www.scopus.com/inward/record.url?scp=85058747989&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058747989&partnerID=8YFLogxK

M3 - Article

VL - 2017

SP - 959

EP - 968

JO - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium

JF - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium

SN - 1559-4076

ER -