Slab and Shrinkage Linear Regression Estimation

This function estimates coefficients in a linear regression model using several shrinkage methods, including Multiplicative Shrinkage, Slab Regression, Linear shrinkage, and Shrinkage Ridge Regression. Each method gives estimators that balance bias and variance by applying shrinkage to the ordinary least squares (OLS) solution. The shrinkage estimators are computed based on different assumptions about the data.

Usage

savvySh(x, y, model_class = c("Multiplicative", "Slab", "Linear", "ShrinkageRR"),
               v = 1, lambda_vals = NULL, nlambda = 100, folds = 10,
               foldid = FALSE, include_Sh = FALSE, exclude = NULL)

Arguments

x

A matrix of predictor variables.

y

A vector of response variable.

model_class

A character string specifying the shrinkage model to use. Options can choose from "Multiplicative", "Slab", "Linear", and "ShrinkageRR". The default is "Multiplicative". If the user supplies more than one model, a warning is issued and only the first option is used.

v

A numeric value controlling the strength of shrinkage for the SR estimator in the "Slab" model. Must be a positive number. Default is 1.

lambda_vals

A vector of lambda values for RR. This is used only when multicollinearity (rank deficiency) is detected and "ShrinkageRR" is not selected. If NULL, a default sequence is used.

nlambda

The number of lambda values to use for cross-validation if lambda_vals is NULL. Only used when multicollinearity is present and "ShrinkageRR" is not called. The default is 100.

folds

Number of folds for cross-validation in RR. This is applicable only if multicollinearity occurs and "ShrinkageRR" is not chosen. The default is 10 and must be an integer >= 3.

foldid

Logical. If TRUE, saves the fold assignments in the output when multicollinearity is detected and "ShrinkageRR" is not used. The default is FALSE.

include_Sh

Logical. If TRUE, includes the Sh estimator in the "Multiplicative" model. The default is FALSE.

exclude

A vector specifying columns to exclude from the predictors. The default is NULL.

Value

A list containing the following elements:

call

The matched function call.

model

The data frame of y and x used in the analysis.

optimal_lambda

If x is full rank, this value is 0. If x is rank-deficient, it is the chosen RR lambda from cross-validation.

model_class

The selected model class.

coefficients

A list of estimated coefficients for each applicable estimator in the model_class.

fitted_values

A list of fitted values for each estimator.

pred_MSE

A list of prediction MSEs for each estimator.

ridge_results (optional)

A list containing detailed results from RR, used when multicollinearity (rank deficiency) is detected in x and the "ShrinkageRR" is not called. This element is included only when RR is applied instead of OLS due to the rank deficiency of x. It contains:

lambda_range

The range of lambda values used in the RR cross-validation.

cvm

A vector of cross-validated MSEs for each lambda in lambda_range.

cvsd

The standard deviation of the cross-validated MSEs for each lambda.

ridge_coefficients

A matrix of coefficients from RR at each lambda value, with each column representing the coefficients corresponding to a specific lambda.

Details

The Slab and Shrinkage Linear Regression Estimation methodology provides four classes of shrinkage estimators that reduce variance in the OLS solution by introducing a small, structured bias. These methods handle overfitting, collinearity, and high-dimensional scenarios by controlling how and where the coefficients are shrunk. Each class offers a distinct strategy for controlling instability and improving mean squared error (MSE) in linear models, tailored for different modeling contexts specified in the model_class argument. Note that if the user provides more than one option in model_class, only the first option is used, and a warning is issued.

Model Classes:

Multiplicative Shrinkage:

This class includes three estimators that use the OLS coefficients as a starting point and apply multiplicative adjustments:

St -

Stein estimator, which shrinks all coefficients toward zero by a single global factor. This aims to reduce MSE while keeping the overall bias fairly uniform across coefficients.

DSh -

Diagonal Shrinkage, assigning an individual factor to each coefficient based on its variance. This yields more targeted shrinkage than the global approach and often achieves a lower MSE.

Sh -

Shrinkage estimator that solves a Sylvester equation for a full (non-diagonal) shrinkage matrix. It is more flexible but also more computationally demanding. Included only if include_Sh = TRUE.

Slab Regression:

Slab Regression applies an adaptive quadratic penalty term to the OLS objective:

SR -

Simple Slab Regression, which modifies the OLS objective by adding a penalty in a fixed direction (often the constant vector). This penalty is controlled by v and does not require cross-validation. It can be viewed as a special case of the generalized lasso but focuses on smooth (quadratic) rather than \(\ell_1\) regularization.

GSR -

Generalized Slab Regression, extending SR by allowing shrinkage along multiple directions. Typically, these directions correspond to the eigenvectors of the design covariance matrix, effectively shrinking principal components.

Linear Shrinkage:

The Linear Shrinkage (LSh) estimator forms a convex combination of the OLS estimator (through the origin) and a target estimator that assumes uncorrelated predictors (diagonal approximation of the covariance). This approach is simpler than a full matrix method and is well-suited for standardized data where the intercept is not needed.

Shrinkage Ridge Regression:

The Shrinkage Ridge Regression (SRR) extends standard RR by shrinking the design covariance matrix toward a spherical target (i.e., a diagonal matrix with equal entries). This additional regularization stabilizes the eigenvalues and yields more robust coefficient estimates, particularly when the predictors lie close to a low-dimensional subspace.

References

Asimit, V., Cidota, M. A., Chen, Z., & Asimit, J. (2025). Slab and Shrinkage Linear Regression Estimation. Retrieved from https://openaccess.city.ac.uk/id/eprint/35005/

Author

Ziwei Chen, Vali Asimit, Marina Anca Cidota, Jennifer Asimit
Maintainer: Ziwei Chen <ziwei.chen.3@citystgeorges.ac.uk>