First order expansion of convex regularized estimators

Pierre C. Bellec, Arun K. Kuchibhotla

Research output: Contribution to journalConference articlepeer-review

Abstract

We consider first order expansions of convex penalized estimators in high-dimensional regression problems with random designs. Our setting includes linear regression and logistic regression as special cases. For a given penalty function h and the corresponding penalized estimator ß^^we construct a quantity ?, the first order expansion of ß, such that the distance between ß and ? is an order of magnitude smaller than the estimation error kß-ß*k. In this sense, the first order expansion ? can be thought of as a generalization of influence functions from the mathematical statistics literature to regularized estimators in high-dimensions. Such first order expansion implies that the risk of ß is asymptotically the same as the risk of ? which leads to a precise characterization of the MSE of ß; this characterization takes a particularly simple form for isotropic design. Such first order expansion also leads to inference results based on ß. We provide sufficient conditions for the existence of such first order expansion for three regularizers: the Lasso in its constrained form, the lasso in its penalized form, and the Group-Lasso. The results apply to general loss functions under some conditions and those conditions are satisfied for the squared loss in linear regression and for the logistic loss in the logistic model.

Original languageEnglish (US)
JournalAdvances in Neural Information Processing Systems
Volume32
StatePublished - 2019
Event33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019 - Vancouver, Canada
Duration: Dec 8 2019Dec 14 2019

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Fingerprint

Dive into the research topics of 'First order expansion of convex regularized estimators'. Together they form a unique fingerprint.

Cite this