Computer vision for dummies » Eigenvectors https://www.visiondummy.com A blog about intelligent algorithms, machine learning, computer vision, datamining and more. Tue, 04 May 2021 14:17:31 +0000 en-US hourly 1 https://wordpress.org/?v=3.8.39 Feature extraction using PCA https://www.visiondummy.com/2014/05/feature-extraction-using-pca/ https://www.visiondummy.com/2014/05/feature-extraction-using-pca/#comments Fri, 16 May 2014 09:33:27 +0000 http://www.visiondummy.com/?p=328 In this article, we discuss how Principal Component Analysis (PCA) works, and how it can be used as a dimensionality reduction technique for classification problems. At the end of this article, Matlab source code is provided for demonstration purposes. In an earlier article, we discussed the so called Curse of Dimensionality and showed that classifiers [...]

The post Feature extraction using PCA appeared first on Computer vision for dummies.

]]> Introduction

In this article, we discuss how Principal Component Analysis (PCA) works, and how it can be used as a dimensionality reduction technique for classification problems. At the end of this article, Matlab source code is provided for demonstration purposes.

In an earlier article, we discussed the so called Curse of Dimensionality and showed that classifiers tend to overfit the training data in high dimensional spaces. The question then rises which features should be preferred and which ones should be removed from a high dimensional feature vector.

If all features in this feature vector were statistically independent, one could simply eliminate the least discriminative features from this vector. The least discriminative features can be found by various greedy feature selection approaches. However, in practice, many features depend on each other or on an underlying unknown variable. A single feature could therefore represent a combination of multiple types of information by a single value. Removing such a feature would remove more information than needed. In the next paragraphs, we introduce PCA as a feature extraction solution to this problem, and introduce its inner workings from two different perspectives.

PCA as a decorrelation method

More often than not, features are correlated. As an example, consider the case where we want to use the red, green and blue components of each pixel in an image to classify the image (e.g. detect dogs versus cats). Image sensors that are most sensitive to red light also capture some blue and green light. Similarly, sensors that are most sensitive to blue and green light also exhibit a certain degree of sensitivity to red light. As a result, the R, G, B components of a pixel are statistically correlated. Therefore, simply eliminating the R component from the feature vector, also implicitly removes information about the G and B channels. In other words, before eliminating features, we would like to transform the complete feature space such that the underlying uncorrelated components are obtained.

Consider the following example of a 2D feature space:

2D Correlated data

Figure 1 2D Correlated data with eigenvectors shown in color.

The features x and y, illustrated by figure 1, are clearly correlated. In fact, their covariance matrix is:

    \begin{equation*} \Sigma = \begin{bmatrix} 16.87 & 14.94 \\[0.3em] 14.94 & 17.27 \\[0.3em] \end{bmatrix} \end{equation*}

In an earlier article we discussed the geometric interpretation of the covariance matrix. We saw that the covariance matrix can be decomposed as a sequence of rotation and scaling operations on white, uncorrelated data, where the rotation matrix is defined by the eigenvectors of this covariance matrix. Therefore, intuitively, it is easy to see that the data D shown in figure 1 can be decorrelated by rotating each data point such that the eigenvectors V become the new reference axes:

(1)   \begin{equation*} D' = V \, D \end{equation*}

2D Uncorrelated data with eigenvectors shown in color.

Figure 2.2D Uncorrelated data with eigenvectors shown in color.

The covariance matrix of the resulting data is now diagonal, meaning that the new axes are uncorrelated:

    \begin{equation*} \Sigma' = \begin{bmatrix} 1.06 & 0.0 \\[0.3em] 0.0 & 16.0 \\[0.3em] \end{bmatrix} \end{equation*}

In fact, the original data used in this example and shown by figure 1 was generated by linearly combining two 1D Gaussian feature vectors x_1 \sim N(0, 1) and x_2 \sim N(0, 1) as follows:

    \begin{align*} x &= x_2 + x_1\\ y &= x_2 - x_1 \end{align*}

Since the features x and y are linear combinations of some unknown underlying components x_1 and x_2, directly eliminating either x or y as a feature would have removed some information from both x_1 and x_2. Instead, rotating the data by the eigenvectors of its covariance matrix, allowed us to directly recover the independent components x_1 and x_2 (up to a scaling factor). This can be seen as follows: The eigenvectors of the covariance matrix of the original data are (each column represents an eigenvector):

    \begin{equation*} V = \begin{bmatrix} -0.7071 & 0.7071 \\[0.3em] 0.7071 & 0.7071 \\[0.3em] \end{bmatrix} \end{equation*}

The first thing to notice is that V in this case is a rotation matrix, corresponding to a rotation of 45 degrees (cos(45)=0.7071), which indeed is evident from figure 1. Secondly, treating V as a linear transformation matrix results in a new coordinate system, such that each new feature x' and y' is expressed as a linear combination of the original features x and y:

(2)   \begin{align*} x' &= -0.7071 \, x + 0.7071 y \\ &= -0.7071 \, (x_2 + x_1) + 0.7071 \, (x_2 - x_1) \\ &= -1.4142 \, x_1 \end{align*}

and

(3)   \begin{align*} y' &= 0.7071 \, x + 0.7071 y \\ &= 0.7071 \, (x_2 + x_1) + 0.7071 \, (x_2 - x_1) y \\ &= 1.4142 \, x_2 \end{align*}

In other words, decorrelation of the feature space corresponds to the recovery of the unknown, uncorrelated components x_1 and y_1 of the data (up to an unknown scaling factor if the transformation matrix was not orthogonal). Once these components have been recovered, it is easy to reduce the dimensionality of the feature space by simply eliminating either x_1 or x_2.

In the above example we started with a two-dimensional problem. If we would like to reduce the dimensionality, the question remains whether to eliminate x_1 (and thus x') or y_1 (and thus y'). Although this choice could depend on many factors such as the separability of the data in case of classification problems, PCA simply assumes that the most interesting feature is the one with the largest variance or spread. This assumption is based on an information theoretic point of view, since the dimension with the largest variance corresponds to the dimension with the largest entropy and thus encodes the most information. The smallest eigenvectors will often simply represent noise components, whereas the largest eigenvectors often correspond to the principal components that define the data.

Dimensionality reduction by means of PCA is then accomplished simply by projecting the data onto the largest eigenvectors of its covariance matrix. For the above example, the resulting 1D feature space is illustrated by figure 3:

2D data projected onto its largest eigenvector

Figure 3. PCA: 2D data projected onto its largest eigenvector.

Obivously, the above example easily generalizes to higher dimensional feature spaces. For instance, in the three-dimensional case, we can either project the data onto the plane defined by the two largest eigenvectors to obtain a 2D feature space, or we can project it onto the largest eigenvector to obtain a 1D feature space. This is illustrated by figure 4:

Principal Component Analysis for 3D data

Figure 4. 3D data projected onto a 2D or 1D linear subspace by means of Principal Component Analysis.

In general, PCA allows us to obtain a linear M-dimensional subspace of the original N-dimensional data, where M \le N. Furthermore, if the unknown, uncorrelated components are Gaussian distributed, then PCA actually acts as an independent component analysis since uncorrelated Gaussian variables are statistically independent. However, if the underlying components are not normally distributed, PCA merely generates decorrelated variables which are not necessarily statistically independent. In this case, non-linear dimensionality reduction algorithms might be a better choice.

PCA as an orthogonal regression method

In the above discussion, we started with the goal of obtaining independent components (or at least uncorrelated components if the data is not normally distributed) to reduce the dimensionality of the feature space. We found that these so called ‘principal components’ are obtained by the eigendecomposition of the covariance matrix of our data. The dimensionality is then reduced by projecting the data onto the largest eigenvectors.

Now let’s forget about our wish to find uncorrelated components for a while. Instead, we will now try to reduce the dimensionality by finding a linear subspace of the original feature space onto which we can project our data such that the projection error is minimized. In the 2D case, this means that we try to find a vector such that projecting the data onto this vector corresponds to a projection error that is lower than the projection error that would be obtained when projecting the data onto any other possible vector. The question is then how to find this optimal vector.

Consider the example shown by figure 5. Three different projection vectors are shown, together with the resulting 1D data. In the next paragraphs, we will discuss how to determine which projection vector minimizes the projection error. Before searching for a vector that minimizes the projection error, we have to define this error function.

Dimensionality reduction by projection onto a linear subspace

Figure 5 Dimensionality reduction by projection onto a linear subspace

A well known method to fit a line to 2D data is least squares regression. Given the independent variable x and the dependent variable y, the least squares regressor corresponds to the line f(x) = ax + b, such that the sum of the squared residual errors \sum_{i=0}^N (f(x_i) - y_i)^2 is minimized. In other words, if x is treated as the independent variable, then the obtained regressor f(x) is a linear function that can predict the dependent variable y such that the squared error is minimal. The resulting model f(x) is illustrated by the blue line in figure 5, and the error that is minimized is illustrated in figure 6.

Linear regression with x as the independent variable

Figure 6. Linear regression where x is the independent variable and y is the dependent variable, corresponds to minimizing the vertical projection error.

However, in the context of feature extraction, one might wonder why we would define feature x as the independent variable and feature y as the dependent variable. In fact, we could easily define y as the independent variable and find a linear function f(y) that predicts the dependent variable x, such that \sum_{i=0}^N (f(y_i) - x_i)^2 is minimized. This corresponds to minimization of the horizontal projection error and results in a different linear model as shown by figure 7:

Linear regression with y as the independent variable

Figure 7. Linear regression where y is the independent variable and x is the dependent variable, corresponds to minimizing the horizontal projection error.

Clearly, the choice of independent and dependent variables changes the resulting model, making ordinary least squares regression an asymmetric regressor. The reason for this is that least squares regression assumes the independent variable to be noise-free, whereas the dependent variable is assumed to be noisy. However, in the case of classification, all features are usually noisy observations such that neither x or y should be treated as independent. In fact, we would like to obtain a model f(x,y) that minimizes both the horizontal and the vertical projection error simultaneously. This corresponds to finding a model such that the orthogonal projection error is minimized as shown by figure 8.

Linear regression where both variables are independent

Figure 8. Linear regression where both variables are independent corresponds to minimizing the orthogonal projection error.

The resulting regression is called Total Least Squares regression or orthogonal regression, and assumes that both variables are imperfect observations. An interesting observation is now that the obtained vector, representing the projection direction that minimizes the orthogonal projection error, corresponds the the largest principal component of the data:

Orthogonal regression based on eigendecomposition

Figure 9. The vector which the data can be projected unto with minimal orthogonal error corresponds to the largest eigenvector of the covariance matrix of the data.

In other words, if we want to reduce the dimensionality by projecting the original data onto a vector such that the squared projection error is minimized in all directions, we can simply project the data onto the largest eigenvectors. This is exactly what we called Principal Component Analysis in the previous section, where we showed that such projection also decorrelates the feature space.

A practical PCA application: Eigenfaces

Although the above examples are limited to two or three dimensions for visualization purposes, dimensionality reduction usually becomes important when the number of features is not negligible compared to the number of training samples. As an example, suppose we would like to perform face recognition, i.e. determine the identity of the person depicted in an image, based on a training dataset of labeled face images. One approach might be to treat the brightness of each pixel of the image as a feature. If the input images are of size 32×32 pixels, this means that the feature vector contains 1024 feature values. Classifying a new face image can then be done by calculating the Euclidean distance between this 1024-dimensional vector, and the feature vectors of the people in our training dataset. The smallest distance then tells us which person we are looking at.

However, operating in a 1024-dimensional space becomes problematic if we only have a few hundred training samples. Furthermore, Euclidean distances behave strangely in high dimensional spaces as discussed in an earlier article. Therefore, we could use PCA to reduce the dimensionality of the feature space by calculating the eigenvectors of the covariance matrix of the set of 1024-dimensional feature vectors, and then projecting each feature vector onto the largest eigenvectors.

Since the eigenvector of 2D data is 2-dimensional, and an eigenvector of 3D data is 3-dimensional, the eigenvectors of 1024-dimensional data is 1024-dimensional. In other words, we could reshape each of the 1024-dimensional eigenvectors to a 32×32 image for visualization purposes. Figure 10 shows the first four eigenvectors obtained by eigendecomposition of the Cambridge face dataset:

Eigenfaces

Figure 10. The four largest eigenvectors, reshaped to images, resulting in so called EigenFaces. (source: https://nl.wikipedia.org/wiki/Eigenface)

Each 1024-dimensional feature vector (and thus each face) can now be projected onto the N largest eigenvectors, and can be represented as a linear combination of these eigenfaces. The weights of these linear combinations determine the identity of the person. Since the largest eigenvectors represent the largest variance in the data, these eigenfaces describe the most informative image regions (eyes, noise, mouth, etc.). By only considering the first N (e.g. N=70) eigenvectors, the dimensionality of the feature space is greatly reduced.

The remaining question is now how many eigenfaces should be used, or in the general case; how many eigenvectors should be kept. Removing too many eigenvectors might remove important information from the feature space, whereas eliminating too few eigenvectors leaves us with the curse of dimensionality. Regrettably there is no straight answer to this problem. Although cross-validation techniques can be used to obtain an estimate of this hyperparameter, choosing the optimal number of dimensions remains a problem that is mostly solved in an empirical (an academic term that means not much more than ‘trial-and-error’) manner. Note that it is often useful to check how much (as a percentage) of the variance of the original data is kept while eliminating eigenvectors. This is done by dividing the sum of the kept eigenvalues by the sum of all eigenvalues.

The PCA recipe

Based on the previous sections, we can now list the simple recipe used to apply PCA for feature extraction:

1) Center the data

In an earlier article, we showed that the covariance matrix can be written as a sequence of linear operations (scaling and rotations). The eigendecomposition extracts these transformation matrices: the eigenvectors represent the rotation matrix, while the eigenvalues represent the scaling factors. However, the covariance matrix does not contain any information related to the translation of the data. Indeed, to represent translation, an affine transformation would be needed instead of a linear transformation.

Therefore, before applying PCA to rotate the data in order to obtain uncorrelated axes, any existing shift needs to be countered by subtracting the mean of the data from each data point. This simply corresponds to centering the data such that its average becomes zero.

2) Normalize the data

The eigenvectors of the covariance matrix point in the direction of the largest variance of the data. However, variance is an absolute number, not a relative one. This means that the variance of data, measured in centimeters (or inches) will be much larger than the variance of the same data when measured in meters (or feet). Consider the example where one feature represents the length of an object in meters, while the second feature represents the width of the object in centimeters. The largest variance, and thus the largest eigenvector, will implicitly be defined by the first feature if the data is not normalized.

To avoid this scale-dependent nature of PCA, it is useful to normalize the data by dividing each feature by its standard deviation. This is especially important if different features correspond to different metrics.

3) Calculate the eigendecomposition

Since the data will be projected onto the largest eigenvectors to reduce the dimensionality, the eigendecomposition needs to be obtained. One of the most widely used methods to efficiently calculate the eigendecomposition is Singular Value Decomposition (SVD).

4) Project the data

To reduce the dimensionality, the data is simply projected onto the largest eigenvectors. Let V be the matrix whose columns contain the largest eigenvectors and let D be the original data whose columns contain the different observations. Then the projected data D' is obtained as D' = V^{\intercal} \, D. We can either choose the number of remaining dimensions, i.e. the columns of V, directly, or we can define the amount of variance of the original data that needs to kept while eliminating eigenvectors. If only N eigenvectors are kept, and e_1...e_N represent the corresponding eigenvalues, then the amount of variance that remains after projecting the original d-dimensional data can be calculated as:

(4)   \begin{equation*} s = \frac{\sum_{i=0}^N e_i}{\sum_{j=0}^d e_j} \end{equation*}

PCA pitfalls

In the above discussion, several assumptions have been made. In the first section, we discussed how PCA decorrelates the data. In fact, we started the discussion by expressing our desire to recover the unknown, underlying independent components of the observed features. We then assumed that our data was normally distributed, such that statistical independence simply corresponds to the lack of a linear correlation. Indeed, PCA allows us to decorrelate the data, thereby recovering the independent components in case of Gaussianity. However, it is important to note that decorrelation only corresponds to statistical independency in the Gaussian case. Consider the data obtained by sampling half a period of y=sin(x):

sinx

Figure 11 Uncorrelated data is only statistically independent if normally distributed. In this example a clear non-linear dependency still exists: y=sin(x).

Although the above data is clearly uncorrelated (on average, the y-value increases as much as it decreases when the x-value goes up) and therefore corresponds to a diagonal covariance matrix, there still is a clear non-linear dependency between both variables.

In general, PCA only uncorrelates the data but does not remove statistical dependencies. If the underlying components are known to be non-Gaussian, techniques such as ICA could be more interesting. On the other hand, if non-linearities clearly exist, dimensionality reduction techniques such as non-linear PCA can be used. However, keep in mind that these methods are prone to overfitting themselves, since more parameters are to be estimated based on the same amount of training data.

A second assumption that was made in this article, is that the most discriminative information is captured by the largest variance in the feature space. Since the direction of the largest variance encodes the most information this is likely to be true. However, there are cases where the discriminative information actually resides in the directions of the smallest variance, such that PCA could greatly hurt classification performance. As an example, consider the two cases of figure 12, where we reduce the 2D feature space to a 1D representation:

PCA might hurt classification performance

Figure 12. In the first case, PCA would hurt classification performance because the data becomes linearly unseparable. This happens when the most discriminative information resides in the smaller eigenvectors.

If the most discriminative information is contained in the smaller eigenvectors, applying PCA might actually worsen the Curse of Dimensionality because now a more complicated classification model (e.g. non-linear classifier) is needed to classify the lower dimensional problem. In this case, other dimensionality reduction methods might be of interest, such as Linear Discriminant Analysis (LDA) which tries to find the projection vector that optimally separates the two classes.

Source Code

The following code snippet shows how to perform principal component analysis for dimensionality reduction in Matlab:
Matlab source code

Conclusion

In this article, we discussed the advantages of PCA for feature extraction and dimensionality reduction from two different points of view. The first point of view explained how PCA allows us to decorrelate the feature space, whereas the second point of view showed that PCA actually corresponds to orthogonal regression.

Furthermore, we briefly introduced Eigenfaces as a well known example of PCA based feature extraction, and we covered some of the most important disadvantages of Principal Component Analysis.

If you’re new to this blog, don’t forget to subscribe, or follow me on twitter!

JOIN MY NEWSLETTER
Receive my newsletter to get notified when new articles and code snippets become available on my blog!
We all hate spam. Your email address will not be sold or shared with anyone else.

The post Feature extraction using PCA appeared first on Computer vision for dummies.

]]> https://www.visiondummy.com/2014/05/feature-extraction-using-pca/feed/ 14 A geometric interpretation of the covariance matrix https://www.visiondummy.com/2014/04/geometric-interpretation-covariance-matrix/ https://www.visiondummy.com/2014/04/geometric-interpretation-covariance-matrix/#comments Thu, 24 Apr 2014 11:09:38 +0000 http://www.visiondummy.com/?p=440 In this article, we provide an intuitive, geometric interpretation of the covariance matrix, by exploring the relation between linear transformations and the resulting data covariance. Most textbooks explain the shape of data based on the concept of covariance matrices. Instead, we take a backwards approach and explain the concept of covariance matrices based on the [...]

The post A geometric interpretation of the covariance matrix appeared first on Computer vision for dummies.

]]> Introduction

In this article, we provide an intuitive, geometric interpretation of the covariance matrix, by exploring the relation between linear transformations and the resulting data covariance. Most textbooks explain the shape of data based on the concept of covariance matrices. Instead, we take a backwards approach and explain the concept of covariance matrices based on the shape of data.


In a previous article, we discussed the concept of variance, and provided a derivation and proof of the well known formula to estimate the sample variance. Figure 1 was used in this article to show that the standard deviation, as the square root of the variance, provides a measure of how much the data is spread across the feature space.

Normal distribution

Figure 1. Gaussian density function. For normally distributed data, 68% of the samples fall within the interval defined by the mean plus and minus the standard deviation.

We showed that an unbiased estimator of the sample variance can be obtained by:

(1)   \begin{align*} \sigma_x^2 &= \frac{1}{N-1} \sum_{i=1}^N (x_i - \mu)^2\\ &= \mathbb{E}[ (x - \mathbb{E}(x)) (x - \mathbb{E}(x))]\\ &= \sigma(x,x) \end{align*}

However, variance can only be used to explain the spread of the data in the directions parallel to the axes of the feature space. Consider the 2D feature space shown by figure 2:

Data with a positive covariance

Figure 2. The diagnoal spread of the data is captured by the covariance.

For this data, we could calculate the variance \sigma(x,x) in the x-direction and the variance \sigma(y,y) in the y-direction. However, the horizontal spread and the vertical spread of the data does not explain the clear diagonal correlation. Figure 2 clearly shows that on average, if the x-value of a data point increases, then also the y-value increases, resulting in a positive correlation. This correlation can be captured by extending the notion of variance to what is called the ‘covariance’ of the data:

(2)   \begin{equation*} \sigma(x,y) = \mathbb{E}[ (x - \mathbb{E}(x)) (y - \mathbb{E}(y))] \end{equation*}

For 2D data, we thus obtain \sigma(x,x), \sigma(y,y), \sigma(x,y) and \sigma(y,x). These four values can be summarized in a matrix, called the covariance matrix:

(3)   \begin{equation*} \Sigma = \begin{bmatrix} \sigma(x,x) & \sigma(x,y) \\[0.3em] \sigma(y,x) & \sigma(y,y) \\[0.3em] \end{bmatrix} \end{equation*}

If x is positively correlated with y, y is also positively correlated with x. In other words, we can state that \sigma(x,y) = \sigma(y,x). Therefore, the covariance matrix is always a symmetric matrix with the variances on its diagonal and the covariances off-diagonal. Two-dimensional normally distributed data is explained completely by its mean and its 2\times 2 covariance matrix. Similarly, a 3 \times 3 covariance matrix is used to capture the spread of three-dimensional data, and a N \times N covariance matrix captures the spread of N-dimensional data.

Figure 3 illustrates how the overall shape of the data defines the covariance matrix:

The spread of the data is defined by its covariance matrix

Figure 3. The covariance matrix defines the shape of the data. Diagonal spread is captured by the covariance, while axis-aligned spread is captured by the variance.

Eigendecomposition of a covariance matrix

In the next section, we will discuss how the covariance matrix can be interpreted as a linear operator that transforms white data into the data we observed. However, before diving into the technical details, it is important to gain an intuitive understanding of how eigenvectors and eigenvalues uniquely define the covariance matrix, and therefore the shape of our data.

As we saw in figure 3, the covariance matrix defines both the spread (variance), and the orientation (covariance) of our data. So, if we would like to represent the covariance matrix with a vector and its magnitude, we should simply try to find the vector that points into the direction of the largest spread of the data, and whose magnitude equals the spread (variance) in this direction.

If we define this vector as \vec{v}, then the projection of our data D onto this vector is obtained as \vec{v}^{\intercal} D, and the variance of the projected data is \vec{v}^{\intercal} \Sigma \vec{v}. Since we are looking for the vector \vec{v} that points into the direction of the largest variance, we should choose its components such that the covariance matrix \vec{v}^{\intercal} \Sigma \vec{v} of the projected data is as large as possible. Maximizing any function of the form \vec{v}^{\intercal} \Sigma \vec{v} with respect to \vec{v}, where \vec{v} is a normalized unit vector, can be formulated as a so called Rayleigh Quotient. The maximum of such a Rayleigh Quotient is obtained by setting \vec{v} equal to the largest eigenvector of matrix \Sigma.

In other words, the largest eigenvector of the covariance matrix always points into the direction of the largest variance of the data, and the magnitude of this vector equals the corresponding eigenvalue. The second largest eigenvector is always orthogonal to the largest eigenvector, and points into the direction of the second largest spread of the data.

Now let’s have a look at some examples. In an earlier article we saw that a linear transformation matrix T is completely defined by its eigenvectors and eigenvalues. Applied to the covariance matrix, this means that:

(4)   \begin{equation*} \Sigma \vec{v} = \lambda \vec{v} \end{equation*}

where \vec{v} is an eigenvector of \Sigma, and \lambda is the corresponding eigenvalue.

If the covariance matrix of our data is a diagonal matrix, such that the covariances are zero, then this means that the variances must be equal to the eigenvalues \lambda. This is illustrated by figure 4, where the eigenvectors are shown in green and magenta, and where the eigenvalues clearly equal the variance components of the covariance matrix.

Eigenvectors of a covariance matrix

Figure 4. Eigenvectors of a covariance matrix

However, if the covariance matrix is not diagonal, such that the covariances are not zero, then the situation is a little more complicated. The eigenvalues still represent the variance magnitude in the direction of the largest spread of the data, and the variance components of the covariance matrix still represent the variance magnitude in the direction of the x-axis and y-axis. But since the data is not axis aligned, these values are not the same anymore as shown by figure 5.

Eigenvectors with covariance

Figure 5. Eigenvalues versus variance

By comparing figure 5 with figure 4, it becomes clear that the eigenvalues represent the variance of the data along the eigenvector directions, whereas the variance components of the covariance matrix represent the spread along the axes. If there are no covariances, then both values are equal.

Covariance matrix as a linear transformation

Now let’s forget about covariance matrices for a moment. Each of the examples in figure 3 can simply be considered to be a linearly transformed instance of figure 6:

White data

Figure 6. Data with unit covariance matrix is called white data.

Let the data shown by figure 6 be D, then each of the examples shown by figure 3 can be obtained by linearly transforming D:

(5)   \begin{equation*} D' = T \, D \end{equation*}

where T is a transformation matrix consisting of a rotation matrix R and a scaling matrix S:

(6)   \begin{equation*} T = R \, S. \end{equation*}

These matrices are defined as:

(7)   \begin{equation*} R = \begin{bmatrix} \cos(\theta) & -\sin(\theta) \\[0.3em] \sin(\theta) & \cos(\theta) \end{bmatrix} \end{equation*}

where \theta is the rotation angle, and:

(8)   \begin{equation*} S = \begin{bmatrix} s_x & 0 \\[0.3em] 0 & s_y \end{bmatrix} \end{equation*}

where s_x and s_y are the scaling factors in the x direction and the y direction respectively.

In the following paragraphs, we will discuss the relation between the covariance matrix \Sigma, and the linear transformation matrix T = R\, S.

Let’s start with unscaled (scale equals 1) and unrotated data. In statistics this is often refered to as ‘white data’ because its samples are drawn from a standard normal distribution and therefore correspond to white (uncorrelated) noise:

Whitened data

Figure 7. White data is data with a unit covariance matrix.

The covariance matrix of this ‘white’ data equals the identity matrix, such that the variances and standard deviations equal 1 and the covariance equals zero:

(9)   \begin{equation*} \Sigma = \begin{bmatrix} \sigma_x^2 & 0 \\[0.3em] 0 & \sigma_y^2 \\ \end{bmatrix} = \begin{bmatrix} 1 & 0 \\[0.3em] 0 & 1 \\ \end{bmatrix} \end{equation*}

Now let’s scale the data in the x-direction with a factor 4:

(10)   \begin{equation*} D' = \begin{bmatrix} 4 & 0 \\[0.3em] 0 & 1 \\ \end{bmatrix} \, D \end{equation*}

The data D' now looks as follows:

Data with variance in the x-direction

Figure 8. Variance in the x-direction results in a horizontal scaling.

The covariance matrix \Sigma' of D' is now:

(11)   \begin{equation*} \Sigma' = \begin{bmatrix} \sigma_x^2 & 0 \\[0.3em] 0 & \sigma_y^2 \\ \end{bmatrix} = \begin{bmatrix} 16 & 0 \\[0.3em] 0 & 1 \\ \end{bmatrix} \end{equation*}

Thus, the covariance matrix \Sigma' of the resulting data D' is related to the linear transformation T that is applied to the original data as follows: D' = T \, D, where

(12)   \begin{equation*} T = \sqrt{\Sigma'} = \begin{bmatrix} 4 & 0 \\[0.3em] 0 & 1 \\ \end{bmatrix}. \end{equation*}

However, although equation (12) holds when the data is scaled in the x and y direction, the question rises if it also holds when a rotation is applied. To investigate the relation between the linear transformation matrix T and the covariance matrix \Sigma' in the general case, we will therefore try to decompose the covariance matrix into the product of rotation and scaling matrices.

As we saw earlier, we can represent the covariance matrix by its eigenvectors and eigenvalues:

(13)   \begin{equation*} \Sigma \vec{v} = \lambda \vec{v} \end{equation*}

where \vec{v} is an eigenvector of \Sigma, and \lambda is the corresponding eigenvalue.

Equation (13) holds for each eigenvector-eigenvalue pair of matrix \Sigma. In the 2D case, we obtain two eigenvectors and two eigenvalues. The system of two equations defined by equation (13) can be represented efficiently using matrix notation:

(14)   \begin{equation*} \Sigma \, V = V \, L \end{equation*}

where V is the matrix whose columns are the eigenvectors of \Sigma and L is the diagonal matrix whose non-zero elements are the corresponding eigenvalues.

This means that we can represent the covariance matrix as a function of its eigenvectors and eigenvalues:

(15)   \begin{equation*} \Sigma = V \, L \, V^{-1} \end{equation*}

Equation (15) is called the eigendecomposition of the covariance matrix and can be obtained using a Singular Value Decomposition algorithm. Whereas the eigenvectors represent the directions of the largest variance of the data, the eigenvalues represent the magnitude of this variance in those directions. In other words, V represents a rotation matrix, while \sqrt{L} represents a scaling matrix. The covariance matrix can thus be decomposed further as:

(16)   \begin{equation*} \Sigma = R \, S \, S \, R^{-1} \end{equation*}

where R=V is a rotation matrix and S=\sqrt{L} is a scaling matrix.

In equation (6) we defined a linear transformation T=R \, S. Since S is a diagonal scaling matrix, S = S^{\intercal}. Furthermore, since R is an orthogonal matrix, R^{-1} = R^{\intercal}. Therefore, T^{\intercal} = (R \, S)^{\intercal} = S^{\intercal} \, R^{\intercal} = S \, R^{-1}. The covariance matrix can thus be written as:

(17)   \begin{equation*} \Sigma = R \, S \, S \, R^{-1} = T \, T^{\intercal}, \end{equation*}

In other words, if we apply the linear transformation defined by T=R \, S to the original white data D shown by figure 7, we obtain the rotated and scaled data D' with covariance matrix T \, T^{\intercal} = \Sigma' = R \, S \, S \, R^{-1}. This is illustrated by figure 10:

The covariance matrix represents a linear transformation of the original data

Figure 10. The covariance matrix represents a linear transformation of the original data.

The colored arrows in figure 10 represent the eigenvectors. The largest eigenvector, i.e. the eigenvector with the largest corresponding eigenvalue, always points in the direction of the largest variance of the data and thereby defines its orientation. Subsequent eigenvectors are always orthogonal to the largest eigenvector due to the orthogonality of rotation matrices.

Conclusion

In this article we showed that the covariance matrix of observed data is directly related to a linear transformation of white, uncorrelated data. This linear transformation is completely defined by the eigenvectors and eigenvalues of the data. While the eigenvectors represent the rotation matrix, the eigenvalues correspond to the square of the scaling factor in each dimension.

If you’re new to this blog, don’t forget to subscribe, or follow me on twitter!

JOIN MY NEWSLETTER
Receive my newsletter to get notified when new articles and code snippets become available on my blog!
We all hate spam. Your email address will not be sold or shared with anyone else.

The post A geometric interpretation of the covariance matrix appeared first on Computer vision for dummies.

]]> https://www.visiondummy.com/2014/04/geometric-interpretation-covariance-matrix/feed/ 47 What are eigenvectors and eigenvalues? https://www.visiondummy.com/2014/03/eigenvalues-eigenvectors/ https://www.visiondummy.com/2014/03/eigenvalues-eigenvectors/#comments Wed, 05 Mar 2014 14:44:53 +0000 http://www.visiondummy.com/?p=111 Eigenvectors and eigenvalues have many important applications in computer vision and machine learning in general. Well known examples are PCA (Principal Component Analysis) for dimensionality reduction or EigenFaces for face recognition. An interesting use of eigenvectors and eigenvalues is also illustrated in my post about error ellipses. Furthermore, eigendecomposition forms the base of the geometric [...]

The post What are eigenvectors and eigenvalues? appeared first on Computer vision for dummies.

]]> Introduction

Eigenvectors and eigenvalues have many important applications in computer vision and machine learning in general. Well known examples are PCA (Principal Component Analysis) for dimensionality reduction or EigenFaces for face recognition. An interesting use of eigenvectors and eigenvalues is also illustrated in my post about error ellipses. Furthermore, eigendecomposition forms the base of the geometric interpretation of covariance matrices, discussed in an more recent post. In this article, I will provide a gentle introduction into this mathematical concept, and will show how to manually obtain the eigendecomposition of a 2D square matrix.

An eigenvector is a vector whose direction remains unchanged when a linear transformation is applied to it. Consider the image below in which three vectors are shown. The green square is only drawn to illustrate the linear transformation that is applied to each of these three vectors.

eigenvectors

Eigenvectors (red) do not change direction when a linear transformation (e.g. scaling) is applied to them. Other vectors (yellow) do.

The transformation in this case is a simple scaling with factor 2 in the horizontal direction and factor 0.5 in the vertical direction, such that the transformation matrix A is defined as:

A=\begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}.

A vector \vec{v}=(x,y) is then scaled by applying this transformation as \vec{v}\prime = A\vec{v}. The above figure shows that the direction of some vectors (shown in red) is not affected by this linear transformation. These vectors are called eigenvectors of the transformation, and uniquely define the square matrix A. This unique, deterministic relation is exactly the reason that those vectors are called ‘eigenvectors’ (Eigen means ‘specific’ in German).

In general, the eigenvector \vec{v} of a matrix A is the vector for which the following holds:

(1)   \begin{equation*} A \vec{v} = \lambda \vec{v} \end{equation*}

where \lambda is a scalar value called the ‘eigenvalue’. This means that the linear transformation A on vector \vec{v} is completely defined by \lambda.

We can rewrite equation (1) as follows:

(2)   \begin{eqnarray*} A \vec{v} - \lambda \vec{v} = 0 \\ \Rightarrow \vec{v} (A - \lambda I) = 0, \end{eqnarray*}

where I is the identity matrix of the same dimensions as A.

However, assuming that \vec{v} is not the null-vector, equation (2) can only be defined if (A - \lambda I) is not invertible. If a square matrix is not invertible, that means that its determinant must equal zero. Therefore, to find the eigenvectors of A, we simply have to solve the following equation:

(3)   \begin{equation*} Det(A - \lambda I) = 0. \end{equation*}

In the following sections we will determine the eigenvectors and eigenvalues of a matrix A, by solving equation (3). Matrix A in this example, is defined by:

(4)   \begin{equation*} A = \begin{bmatrix} 2 & 3 \\ 2 & 1 \end{bmatrix}. \end{equation*}

Calculating the eigenvalues

To determine the eigenvalues for this example, we substitute A in equation (3) by equation (4) and obtain:

(5)   \begin{equation*} Det\begin{pmatrix}2-\lambda&3\\2&1-\lambda\end{pmatrix}=0. \end{equation*}

Calculating the determinant gives:

(6)   \begin{align*} &(2-\lambda)(1-\lambda) - 6 = 0\\ \Rightarrow &2 - 2 \lambda - \lambda - \lambda^2 -6 = 0\\ \Rightarrow &{\lambda}^2 - 3 \lambda -4 = 0. \end{align*}

To solve this quadratic equation in \lambda, we find the discriminant:

    \begin{equation*} D = b^2 -4ac = (-3)^2 -4*1*(-4) = 9+16 = 25. \end{equation*}

Since the discriminant is strictly positive, this means that two different values for \lambda exist:

(7)   \begin{align*} \lambda _1 &= \frac{-b - \sqrt{D}}{2a} = \frac{3-5}{2} = -1,\\ \lambda _2 &= \frac{-b + \sqrt{D}}{2a} = \frac{3+5}{2} = 4. \end{align*}

We have now determined the two eigenvalues \lambda_1 and \lambda_2. Note that a square matrix of size N \times N always has exactly N eigenvalues, each with a corresponding eigenvector. The eigenvalue specifies the size of the eigenvector.

Calculating the first eigenvector

We can now determine the eigenvectors by plugging the eigenvalues from equation (7) into equation (1) that originally defined the problem. The eigenvectors are then found by solving this system of equations.

We first do this for eigenvalue \lambda_1, in order to find the corresponding first eigenvector:

    \begin{equation*} \begin{bmatrix}2&3\\2&1\end{bmatrix} \begin{bmatrix}x_{11}\\x_{12}\end{bmatrix} = -1 \begin{bmatrix}x_{11}\\x_{12}\end{bmatrix}. \end{equation*}

Since this is simply the matrix notation for a system of equations, we can write it in its equivalent form:

(8)   \begin{eqnarray*} \left\{ \begin{array}{lr} 2x_{11} + 3x_{12} = -x_{11}\\ 2x_{11} + x_{12} = -x_{12} \end{array} \right. \end{eqnarray*}

and solve the first equation as a function of x_{12}, resulting in:

(9)   \begin{equation*} x_{11} = -x_{12}. \end{equation*}

Since an eigenvector simply represents an orientation (the corresponding eigenvalue represents the magnitude), all scalar multiples of the eigenvector are vectors that are parallel to this eigenvector, and are therefore equivalent (If we would normalize the vectors, they would all be equal). Thus, instead of further solving the above system of equations, we can freely chose a real value for either x_{11} or x_{12}, and determine the other one by using equation (9).

For this example, we arbitrarily choose x_{12} = 1, such that x_{11}=-1. Therefore, the eigenvector that corresponds to eigenvalue \lambda_1 = -1 is

(10)   \begin{equation*} \vec{v}_1 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}. \end{equation*}

Calculating the second eigenvector

Calculations for the second eigenvector are similar to those needed for the first eigenvector;
We now substitute eigenvalue \lambda_2=4 into equation (1), yielding:

(11)   \begin{equation*} \begin{bmatrix}2&3\\2&1\end{bmatrix} \begin{bmatrix}x_{21}\\x_{22}\end{bmatrix} = 4 * \begin{bmatrix}x_{21}\\x_{22}\end{bmatrix}. \end{equation*}

Written as a system of equations, this is equivalent to:

(12)   \begin{eqnarray*} \left\{ \begin{array}{lr} 2x_{21} + 3x_{22} = 4x_{21}\\ 2x_{21} + x_{22} = 4x_{22} \end{array} \right. \end{eqnarray*}

Solving the first equation as a function of x_{21} resuls in:

(13)   \begin{equation*} x_{22} = \frac{3}{2}x_{21} \end{equation*}

We then arbitrarily choose x_{21}=2, and find x_{22}=3. Therefore, the eigenvector that corresponds to eigenvalue \lambda_2 = 4 is

(14)   \begin{equation*} \vec{v}_2 = \begin{bmatrix} 3 \\ 2 \end{bmatrix}. \end{equation*}

Conclusion

In this article we reviewed the theoretical concepts of eigenvectors and eigenvalues. These concepts are of great importance in many techniques used in computer vision and machine learning, such as dimensionality reduction by means of PCA, or face recognition by means of EigenFaces.

If you’re new to this blog, don’t forget to subscribe, or follow me on twitter!

JOIN MY NEWSLETTER
Receive my newsletter to get notified when new articles and code snippets become available on my blog!
We all hate spam. Your email address will not be sold or shared with anyone else.

The post What are eigenvectors and eigenvalues? appeared first on Computer vision for dummies.

]]> https://www.visiondummy.com/2014/03/eigenvalues-eigenvectors/feed/ 20