# Principal component regression

Template:Regression bar Principal component regression (PCR) is a particular regression analysis technique in statistics that is based on principal component analysis (PCA). Typically, it considers regressing the outcome (also known as the response or, the dependent variable) on a set of covariates (also known as predictors or, explanatory variables or, independent variables) based on a standard linear regression model, but uses PCA for estimating the unknown regression coefficients in the model.

In PCR, instead of regressing the dependent variable on the explanatory variables directly, the principal components of the explanatory variables are used as regressors. One typically uses only a subset of all the principal components for regression, thus making PCR some kind of a regularized procedure. Often the principal components with higher variances (the ones based on eigenvectors corresponding to the higher eigenvalues of the sample variance-covariance matrix of the explanatory variables) are selected as regressors. However, for the purpose of predicting the outcome, the principal components with low variances may also be important, in some cases even more important. 

One major use of PCR lies in overcoming the multicollinearity problem which arises when two or more of the explanatory variables are close to being collinear. PCR can aptly deal with such situations by excluding some of the low-variance principal components in the regression step. In addition, by usually regressing on only a subset of all the principal components, PCR can result in dimension reduction through substantially lowering the effective number of parameters characterizing the underlying model. This can be particularly useful in settings with high dimensional covariates. Also, through appropriate selection of the principal components to be used for regression, PCR can lead to efficient prediction of the outcome based on the assumed model.

## The Principle

The PCR method may be broadly divided into three major steps:

1. $\;\;$ Perform PCA on the observed data matrix for the explanatory variables to obtain the principal components, and then (usually) select a subset, based on some appropriate criteria, of the principal components so obtained for further use.
2. $\;\;$ Now regress the observed vector of outcomes on the selected principal components as covariates, using ordinary least squares regression (linear regression) to get a vector of estimated regression coefficients (with dimension equal to the number of selected principal components).
3. $\;\;$ Now transform this vector back to the scale of the actual covariates, using the selected PCA loadings (the eigenvectors corresponding to the selected principal components) to get the final PCR estimator (with dimension equal to the total number of covariates) for estimating the regression coefficients characterizing the original model.

## Fundamental Characteristics and Applications of the PCR Estimator

### Two Basic Properties

The fitting process for obtaining the PCR estimator involves regressing the response vector on the derived data matrix $W_{k}$ which has orthogonal columns for any $k\in \{1,...,p\}$ since the principal components are mutually orthogonal to each other. Thus in the regression step, performing a multiple linear regression jointly on the $k$ selected principal components as covariates is equivalent to carrying out $k$ independent simple linear regressions (or, univariate regressions) separately on each of the $k$ selected principal components as a covariate.

### Variance Reduction

Thus, for any $k\in \{1,...,p\},{\text{Var}}({\widehat {\boldsymbol {\beta }}}_{ols})-{\text{Var}}({\widehat {\boldsymbol {\beta }}}_{k})\succeq 0$ where $A\succeq 0$ indicates that a square symmetric matrix $A$ is non-negative definite. Consequently, any given linear form of the PCR estimator has a lower variance compared to that of the same linear form of the ordinary least squares estimator.

Under multicollinearity, two or more of the covariates are highly correlated, so that one can be linearly predicted from the others with a non-trivial degree of accuracy. Consequently, the columns of the data matrix $\mathbf {X}$ that correspond to the observations for these covariates tend to become linearly dependent and therefore, $\mathbf {X}$ tends to become rank deficient losing its full column rank structure. More quantitatively, one or more of the smaller eigenvalues of $\mathbf {X} ^{T}\mathbf {X}$ get(s) very close or, become(s) exactly equal to $0$ under such situations. The variance expressions above indicate that these small eigenvalues have the maximum inflation effect on the variance of the least squares estimator, thereby destabilizing the estimator significantly when they are close to $0$ . This issue can be effectively addressed through using a PCR estimator obtained by excluding the principal components corresponding to these small eigenvalues.

### Dimension Reduction

Thus any potential dimension reduction may be achieved by choosing $k$ , the number of principal components to be used, through appropriate thresholding on the cumulative sum of the eigenvalues of $\mathbf {X} ^{T}\mathbf {X}$ . Since the smaller eigenvalues do not contribute significantly to the cumulative sum, the corresponding principal components may be continued to be dropped as long as the desired threshold limit is not exceeded. The same criteria may also be used for addressing the multicollinearity issue whereby the principal components corresponding to the smaller eigenvalues may be ignored as long as the threshold limit is maintained.

### Regularization Effect

Since the PCR estimator typically uses only a subset of all the principal components for regression, it can be viewed as some sort of a regularized procedure. More specifically, for any $1\leq k , the PCR estimator ${\widehat {\boldsymbol {\beta }}}_{k}$ denotes the regularized solution to the following constrained minimization problem:

Thus, when only a proper subset of all the principal components are selected for regression, the PCR estimator so obtained is based on a hard form of regularization that constrains the resulting solution to the column space of the selected principal component directions, and consequently restricts it to be orthogonal to the excluded directions.

### Optimality of PCR among a Class of Regularized Estimators

Given the constrained minimization problem as defined above, let us consider the following generalized version of it:

### Efficiency

In order to ensure efficient estimation and prediction performance of PCR as an estimator of ${\boldsymbol {\beta }}$ , Park (1981)  proposes the following guideline for selecting the principal components to be used for regression: Drop the $j^{th}$ principal component if and only if $\lambda _{j}<(p\sigma ^{2})/{\boldsymbol {\beta }}^{T}{\boldsymbol {\beta }}$ . Practical implementation of this guideline of course requires estimates for the unknown model parameters $\sigma ^{2}$ and ${\boldsymbol {\beta }}$ . In general, they may be estimated using the unrestricted least squares estimates obtained from the original full model. Park (1981)  however provides a slightly modified set of estimates that may be better suited for this purpose.

Unlike the criteria based on the cumulative sum of the eigenvalues of $\mathbf {X} ^{T}\mathbf {X}$ , which is probably more suited for addressing the multicollinearity problem and for performing dimension reduction, the above criteria actually attempts to improve the prediction and estimation efficiency of the PCR estimator by involving both the outcome as well as the covariates in the process of selecting the principal components to be used in the regression step. Alternative approaches with similar goals include selection of the principal components based on cross-validation or the Mallows's Cp criteria. Often, the principal components are also selected based on their degree of association with the outcome.

### Shrinkage Effect of PCR

In general, PCR is essentially a shrinkage estimator that usually retains the high variance principal components (corresponding to the higher eigenvalues of $\mathbf {X} ^{T}\mathbf {X}$ ) as covariates in the model and discards the remaining low variance components (corresponding to the lower eigenvalues of $\mathbf {X} ^{T}\mathbf {X}$ ). Thus it exerts a discrete shrinkage effect on the low variance components nullifying their contribution completely in the original model. In contrast, the ridge regression estimator exerts a smooth shrinkage effect through the regularization parameter (or, the tuning parameter) inherently involved in its construction. While it does not completely discard any of the components, it exerts a shrinkage effect over all of them in a continuous manner so that the extent of shrinkage is higher for the low variance components and lower for the high variance components. Frank and Friedman (1993) conclude that for the purpose of prediction itself, the ridge estimator, owing to its smooth shrinkage effect, is perhaps a better choice compared to the PCR estimator having a discrete shrinkage effect.

In addition, the principal components are obtained from the eigen-decomposition of $\mathbf {X}$ that involves the observations for the explanatory variables only. Therefore the resulting PCR estimator obtained from using these principal components as covariates need not necessarily have satisfactory predictive performance for the outcome. A somewhat similar estimator that tries to address this issue through its very construction is the partial least squares (PLS) estimator. Similar to PCR, PLS also uses derived covariates of lower dimensions. However unlike PCR, the derived covariates for PLS are obtained based on using both the outcome as well as the covariates. While PCR seeks the high variance directions in the space of the covariates, PLS seeks the directions in the covariate space that are most useful for the prediction of the outcome.

Recently, a variant of the classical PCR known as the Supervised PCR was proposed by Bair, Hastie, Paul and Tibshirani (2006). In a spirit similar to that of PLS, it attempts at obtaining derived covariates of lower dimensions based on a criteria that involves both the outcome as well as the covariates. The method starts by performing a set of $p$ simple linear regressions (or, univariate regressions) wherein the outcome vector is regressed separately on each of the $p$ covariates taken one at a time. Then, for some $m\in \{1,..,p\}$ , the first $m$ covariates that turn out to be the most correlated with the outcome (based on the degree of significance of the corresponding estimated regression coefficients) are selected for further use. A conventional PCR, as described earlier, is then performed, but now it is based on only the $n\times m$ data matrix corresponding to the observations for the selected covariates. The number of covariates used: $m\in \{1,..,p\}$ and the subsequent number of principal components used: $k\in \{1,..,m\}$ are usually selected by cross-validation.

## Generalization to Kernel Settings

The classical PCR method as described above is based on classical PCA and considers a linear regression model for predicting the outcome based on the covariates. However, it can be easily generalized to a kernel machine setting whereby the regression function need not necessarily be linear in the covariates, but instead it can belong to the Reproducing Kernel Hilbert Space associated with any arbitrary (possibly non-linear), symmetric positive-definite kernel. The linear regression model turns out to be a special case of this setting when the kernel function is chosen to be the linear kernel.

In general, under the kernel machine setting, the vector of covariates is first mapped into a high dimensional (potentially infinite dimensional) feature space characterized by the kernel function chosen. The mapping so obtained is known as the feature map and each of its coordinates, also known as the feature elements, corresponds to one feature (may be linear or, non-linear) of the covariates. The regression function is then assumed to be a linear combination of these feature elements. Thus, the underlying regression model in the kernel machine setting is essentially a linear regression model with the understanding that instead of the original set of covariates, the predictors are now given by the vector (potentially infinite dimensional) of feature elements obtained by transforming the actual covariates using the feature map.

However, the kernel trick actually enables us to operate in the feature space without ever explicitly computing the feature map. It turns out that it is only sufficient to compute the pairwise inner products among the feature maps for the observed covariate vectors and these inner products are simply given by the values of the kernel function evaluated at the corresponding pairs of covariate vectors. The pairwise inner products so obtained may therefore be represented in the form of a $n\times n$ symmetric non-negative definite matrix also known as the kernel matrix.

PCR in the kernel machine setting can now be implemented by first appropriately centering this kernel matrix (K, say) with respect to the feature space and then performing a kernel PCA on the centered kernel matrix (K', say) whereby an eigendecomposition of K' is obtained. Kernel PCR then proceeds by (usually) selecting a subset of all the eigenvectors so obtained and then performing a standard linear regression of the outcome vector on these selected eigenvectors. The eigenvectors to be used for regression are usually selected using cross-validation. The estimated regression coefficients (having the same dimension as the number of selected eigenvectors) along with the corresponding selected eigenvectors are then used for predicting the outcome for a future observation. In machine learning, this technique is also known as spectral regression.

Clearly, kernel PCR has a discrete shrinkage effect on the eigenvectors of K', quite similar to the discrete shrinkage effect of classical PCR on the principal components, as discussed earlier. However, it should be noted that the feature map associated with the chosen kernel could potentially be infinite dimensional, and hence the corresponding principal components and principal component directions could be infinite dimensional as well. Therefore these quantities are often practically intractable under the kernel machine setting. Kernel PCR essentially works around this problem by considering an equivalent dual formulation based on using the spectral decomposition of the associated kernel matrix. Under the linear regression model (which corresponds to choosing the kernel function as the linear kernel), this amounts to considering a spectral decomposition of the corresponding $n\times n$ kernel matrix $\mathbf {X} \mathbf {X} ^{T}$ and then regressing the outcome vector on a selected subset of the eigenvectors of $\mathbf {X} \mathbf {X} ^{T}$ so obtained. It can be easily shown that this is the same as regressing the outcome vector on the corresponding principal components (which are finite dimensional in this case), as defined in the context of the classical PCR. Thus, for the linear kernel, the kernel PCR based on a dual formulation is exactly equivalent to the classical PCR based on a primal formulation. However, for arbitrary (and possibly non-linear) kernels, this primal formulation may become intractable owing to the infinite dimensionality of the associated feature map. Thus classical PCR becomes practically infeasible in that case, but kernel PCR based on the dual formulation still remains valid and computationally scalable.