A split-and-conquer approach for analysis of extraordinarily large data

TY - JOUR

T1 - A split-and-conquer approach for analysis of extraordinarily large data

AU - Chen, Xueying

AU - Xie, Min Ge

PY - 2014/10

Y1 - 2014/10

N2 - If there are datasets, too large to fit into a single computer or too expensive for a computationally intensive data analysis, what should we do? We propose a split-and-conquer approach and illustrate it using several computationally intensive penalized regression methods, along with a theoretical support. We show that the split-and-conquer approach can substantially reduce computing time and computer memory requirements. The proposed methodology is illustrated numerically using both simulation and data examples.

AB - If there are datasets, too large to fit into a single computer or too expensive for a computationally intensive data analysis, what should we do? We propose a split-and-conquer approach and illustrate it using several computationally intensive penalized regression methods, along with a theoretical support. We show that the split-and-conquer approach can substantially reduce computing time and computer memory requirements. The proposed methodology is illustrated numerically using both simulation and data examples.

KW - Big data

KW - Combining results from independent analyses

KW - Distributed computing

KW - Generalized linear models

KW - Large sample theory

KW - Penalized regression

UR - https://www.scopus.com/pages/publications/84946566592

UR - https://www.scopus.com/pages/publications/84946566592#tab=citedBy

U2 - 10.5705/ss.2013.088

DO - 10.5705/ss.2013.088

M3 - Article

SN - 1017-0405

VL - 24

SP - 1655

EP - 1684

JO - Statistica Sinica

JF - Statistica Sinica

IS - 4

ER -