Positive Definite, Positive Semidefinite Covariance Matrix

Explained: nonsingular random vector positive definite matrix positive semidefinite matrix singular positive semidefinite matrix singular random vector

A real symmetric matrix h is positive semidefinite if 0 for all row vectors b. This definition may seem abstruse, but positive semidefinite matrices have a number of interesting properties. One of these is that all the eigenvalues of a positive semidefinite matrix are real and nonnegative.

Any positive semidefinite matrix h can be factored in the form for some real square matrix k, which we may think of as a matrix square root of h. The matrix k is not unique, so multiple factorizations of a given matrix h are possible. This is analogous to the fact that square roots of positive numbers are not unique either. A standard method for factoring positive semidefinite matrices is the Cholesky factorization.

Positive semidefinite matrices are important in probability theory because covariance matrices are always positive semidefinite—so all properties of positive semidefinite matrices are properties of covariance matrices. Because a correlation matrix is essentially a normalized covariance matrix, results apply equally to them.

Positive semidefinite matrices fall into two categories: those that are singular and those that are non-singular.

The former don't have a specific name, so we simply call them singular positive semidefinite matrices. In addition to the condition that 0 for all row vectors b, it is true of singular positive semidefinite matrices that that = 0 for some row vector b 0. As with all singular matrices, singular positive semidefinite matrices have at least one eigenvalue that equals 0.

Non-singular positive semidefinite matrices are called positive definite matrices. They satisfy the condition that > 0 for all row vectors b 0. All their eigenvalues are real and positive.

A random vector is called singular if its covariance matrix is singular (hence singular positive semidefinite). It is called non-singular if its covariance matrix is non-singular (hence positive definite).

Suppose random vector X is singular with covariance matrix . Because is singular, there exists a row vector b 0 such that bb' = 0. Consider the random variable bX. Then (see the article linear polynomial of a random vector)

Since our random variable bX has 0 variance, it must equal (with probability 1) some constant a. This argument is reversible, so we conclude that a random vector X is singular if and only if there exists a row vector b 0 and a constant a such that

Dispensing with matrix notation, this becomes

Since b 0, at lease one component is nonzero. Without loss of generality, assume 0. Rearranging [3], we obtain

which expresses component X₁ as a linear polynomial of the other components X_i. We conclude that a random vector X is singular if and only if one of its components is a linear polynomial of the other components. In this sense, a singular covariance matrix indicates that at least one component of a random vector is extraneous.

If one component of X is a linear polynomial of the rest, then all realizations of X must fall in a plane within . The random vector X can be thought of as an m-dimensional random vector (m < n) sitting in a plane within . This is illustrated with realizations of a singular two-dimensional random vector X in Exhibit 1.

Realizations of a Singular Random Vector Exhibit 1

Realizations of a singular 2-dimensional random vector are illustrated.

If a random vector X is singular, but the plane it sits in is not aligned with the coordinate system of , we may not immediately realize that it is singular from its covariance matrix . A simple test for singularity is to calculate the determinant | | of the covariance matrix. If this equals 0, X is singular.

Once we know that X is singular, we can apply a change of variables to eliminate extraneous components X_i and transform X into an equivalent m-dimensional random vector Y, m < n. The change of variables will do this by transforming (rotating, shifting, etc.) the plane that realizations of X sit in so that it aligns with the coordinate system of . Such a change of variables is obtained with a linear polynomial of the form:

Consider a three-dimensional random vector X with mean vector and covariance matrix

We note that has determinant | | = 0, so it is singular. We propose to transform X into an equivalent 2-dimensional random vector Y using a linear polynomial of the form [5]. For convenience, let's find a transformation such that Y will have mean vector 0 and covariance matrix I:

We first solve for k. We know (see the article linear polynomial of a random vector)

so we seek a factorization . Applying the Cholesky factorization and discarding an extraneous column of 0's, we obtain:

Solving next for d:

Accordingly, our transformation is

Exhibit 2 illustrates how this change of variables transforms the plane in which X sits so that it aligns with the coordinate system of .

Example: Transforming a Singular 3-Dimensional Random Vector into an Equivalent Non-Singular 2-Dimensional Random Vector Exhibit 2

Our change of variables transforms the plane that realizations of X sit in so that it aligns with the coordinate system. The third extraneous component of X "drops out.”

Exercises
Below are described four three-dimensional random vectors: W, V, X, and Y. Assuming their second moments exist, which of the random vectors has a singular covariance matrix? a. Components V1 and V2 are independent. Component V3 = 2V1 – 5V2 + 1. b. Components W1 and W2 are independent. Component W3 = W1 – log(W2). c. Components X1, X2, and X3 represent next year's total returns for three different companies' common stocks. d. Components Y1 and Y2 represent tomorrow's prices for the nearby 3-month Treasury bill and 3-month Eurodollar futures. Component Y3 represents tomorrow's price difference between those two futures. [solution] True or false: a. A covariance matrix is singular if and only if it is positive definite. b. A covariance matrix is nonsingular if and only if it is positive semidefinite. c. Every random vector has a positive semidefinite covariance matrix. [solution] Consider a singular random vector X with mean vector and covariance matrix [e1] Transform X into an equivalent two-dimensional random vector Y with mean vector 0 and covariance matrix I: [e2] [solution]
Sponsored Links
Related Internal Links
Cholesky matrix A lower-triangular matrix that acts as a matrix "square root" for a positive definite matrix. correlation A parameter that indicates the tendency for two random variables to "move together" of "co-vary." eigenvalue, eigenvector Concepts from linear algebra. gradient, Hessian, Jacobian Multidimensional generalizations of first and second derivatives. joint normal distribution A multivariate distribution with normal marginal distributions. linear polynomial of a random vector A random variable or random vector that is defined as a linear polynomial of a random vector. multicollinear A covariance matrix is muticollinear if it is "almost" singular. principal component analysis A technique for orthogonalizing a random vector.
Related Forum Discussions
Positive definiteness of Correlation Matrix 29 Jul 2005 Intuitively, why must a correlation matrix be positive definite or positive semidefinite?
http://www.riskglossary.com copyright © Glyn A. Holton, 2006 Although the information in this website has been presented with care and obtained from sources the author believes to be reliable, there is no guarantee that it is accurate. Such information may be incomplete, condensed, outdated or presented with errors. The content of the website is for information purposes only. It is provided gratuitously, so the author shall not be liable under any theory for any damages suffered by any user. The author does not provide investment advice, and this website is not a vehicle for communicating investment advice.