Shrinkage and absolute penalty estimation in linear regression models

Abstract

In predicting a response variable using multiple linear regression model, several candidate models may be available which are subsets of the full model. Shrinkage estimators borrow information from the full model and provides a hybrid estimate of the regression parameters by shrinking the full model estimates toward the candidate submodel. The process introduces bias in the estimation but reduces the overall prediction error that offsets the bias. In this article, we give an overview of shrinkage estimators and their asymptotic properties. A real data example is given and a Monte Carlo simulation study is carried out to evaluate the performance of shrinkage estimators compared to the absolute penalty estimators such as least absolute shrinkage and selection operator (LASSO), adaptive LASSO and smoothly clipped absolute deviation (SCAD) based on prediction errors criterion in a multiple linear regression setup. WIREs Comput Stat 2012, 4:541–553. DOI: 10.1002/wics.1232

Keywords
shrinkage estimation; absolute penalty estimation; LASSO; adaptive LASSO; SCAD

How to cite

S. Ejaz Ahmed, and S. E. Raheem, (2012). Shrinkage and absolute penalty estimation in linear modelsWIREs Computational StatisticsVolume 4, Issue 6, pages 541–553, November/December 2012.

Absolute penalty and shrinkage estimation in partially linear models

Abstract
In the context of a partially linear regression model, shrinkage semiparametric estimation is considered based on the Stein-rule. In this framework, the coefficient vector is partitioned into two sub-vectors: the first sub-vector gives the coefficients of interest, i.e., main effects (for example, treatment effects), and the second sub-vector is for variables that may or may not need to be controlled. When estimating the first sub-vector, the best estimate may be obtained using either the full model that includes both sub-vectors, or the reduced model which leaves out the second sub-vector. It is demonstrated that shrinkage estimators which combine two semiparametric estimators computed for the full model and the reduced model outperform the semiparametric estimator for the full model. Using the semiparametric estimate for the reduced model is best when the second sub-vector is the null vector, but this estimator suffers seriously from bias otherwise. The relative dominance picture of suggested estimators is investigated. In particular, suitability of estimating the nonparametric component based on the B-spline basis function is explored. Further, the performance of the proposed estimators is compared with an absolute penalty estimator through Monte Carlo simulation. Lasso and adaptive lasso were implemented for simultaneous model selection and parameter estimation. A real data example is given to compare the proposed estimators with lasso and adaptive lasso estimators.

Keywords
Partially linear model; James–Stein estimator; Absolute penalty estimation; Lasso; Adaptive lasso; B-spline approximation; Semiparametric model; Monte Carlo simulation

How to cite

Raheem, S. E, Ahmed S. E., Doksum K. A. (2012). Absolute penalty and shrinkage estimation in partially linear models. Computational Statistics and Data Analysis. 56(4):874-891.