Multicollinear

<body onLoad="MM_checkPlugin('Shockwave Flash','','http://www.riskglossary.com/default_nofl.htm',false);return document.MM_returnValue" stylesrc="../_private/template.htm" bgcolor="#FFFFFF" link="#669933" vlink="#000000" alink="#000000"> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" id="table1"> <tr> <td width="740"> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" height="37" id="table2"> <tr> <td height="35" width="100%"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" id="table3"> <tr> <td width="100%" align="left" valign="top"> <p class="header_gold">Multicollinear</td> <td width="45" align="right" valign="bottom"> <map name="FPMap6"> <area href="http://www.contingencyanalysis.com" shape="rect" coords="1, 0, 44, 11" target="_top"> </map> <img border="0" src="../images/home.gif" usemap="#FPMap6" width="45" height="12"></td> </tr> </table> </td> </tr> <tr> <td width="100%" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td width="0%" valign="top" bgcolor="#FFFFCC"> <table border="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" cellspacing="1" id="table4"> <tr> <td width="0%" align="left" valign="top"> <p class="header2_black">Explained:<br> <img border="0" src="../images/blank_1x100.gif" width="100" height="1"></td> <td width="0%" align="left" valign="top"><p class="word_list"> <a href="#multicollinear">multicollinear</a><br> <img border="0" src="../images/blank_1x160.gif" width="160" height="1"></td> <td width="100%" align="left" valign="top"><p class="word_list"> <br> <img border="0" src="../images/blank_1x160.gif" width="160" height="1"></td> </tr> </table> </td> </tr> </table> <table border="0" cellpadding="0" style="border-collapse: collapse" bordercolor="#111111" width="310" align="right" id="table5"> <tr> <td align="right" width="9" height="14"></td> <td width="300" height="14"> <font size="2"> </font><font size="4"> </font></td> </tr> <tr> <td align="right" width="9"> </td> <td width="300"> <script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script>  </td> </tr> </table> <p class="body">A random vector is <b><a name="multicollinear"> multicollinear</a></b> if it is "almost" <a href="../articles/positive_definite_matrix.htm">singular</a>. Let's consider an example from finance.</p>  <p class="body">Suppose we are analyzing the <a href="../articles/market_risk.htm">market risk</a> in a natural gas trading portfolio. Random variables represent tomorrow’s values for each price the portfolio is exposed to. The portfolio holds New York Mercantile Exchange (NYMEX) Henry Hub futures out to 24 months, so there are 24 <a href="../articles/future.htm">futures</a> prices. It also has <a href="../articles/forward.htm">forward</a> positions out to 18 months for 30 delivery points, for another 540 prices. In total, our model depends upon a vector of 564 random variables! </p>  <p class="body">Based upon a <a href="../articles/time_series_stochastic_process.htm">time series analysis</a> of historical price data, we construct a 564<img border="0" src="../symbols/cross.gif" align="absbottom" width="11" height="13">564 <a href="../articles/correlation.htm">covariance matrix</a> for our random vector of prices. Gazing at the 318,096 <a href="../articles/standard_deviation.htm">variances</a> and <a href="../articles/correlation.htm">covariances</a> of our matrix, we wonder: Do we really need all these numbers? </p> <p class="body">Intuitively, we know that the random variables are interdependent. Prices for 6-month and 7-month Transco Zone 2 delivery are highly correlated. So are 3-month prices for adjacent Transco Zones 1 and 2. Because of such interdependencies, it is conceivable that our random vector is <a href="../articles/positive_definite_matrix.htm">singular</a>, but this is probably not the case. Singularity arises infrequently in applications. A more common situation is “almost” singularity, which is known as multicollinearity. </p> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="345" align="left" id="table6"> <tr> <td width="168" height="15"> <p class="ad_legend"> <a target="_top" href="http://www.riskexpertise.com/link/advertising.htm"> <font color="#000000">Ads by Contingency Analysis</font></a></td> <td width="168" height="15" align="right"> <p class="ad_legend"> <a target="_top" href="http://www.riskexpertise.com/link/advertising.htm"> <font color="#000000">Advertise on this site</font></a></td> <td width="9" rowspan="2"> </td> </tr> <tr> <td width="336" colspan="2">  <iframe src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=84506;type=iframe" allowtransparency="true" background-color="transparent" style="margin: 0px; padding: 0px; border-width: 0px" width="336" height="265" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" name="I1"> <![if lt IE 4]> <script src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=84506;type=js" type="text/javascript"> </script> <noscript> <a href="http://servedbyadbutler.com/go2/;ID=147917;size=336x265;setID=84506"> <img src="http://servedbyadbutler.com/adserve.ibs/;ID=147917;size=336x265;setID=84506;type=img" border="0" height="265" width="336" alt=""></a> </noscript><![endif]></iframe>  </td> </tr> </table> <p class="body">We illustrate with two four-dimensional random vectors. Random vector <i><b>X</b></i> is singular. Its first three components <img border="0" src="../symbols/X1.gif" align="absbottom" width="15" height="15">, <img border="0" src="../symbols/X2.gif" align="absbottom" width="16" height="15"> and <img border="0" src="../symbols/X3.gif" align="absbottom" width="16" height="15"> are uncorrelated, each with <a href="../articles/mean.htm">mean</a> 0 and <a href="../articles/standard_deviation.htm">standard deviation</a> 1. The fourth component <img border="0" src="../symbols/X4.gif" align="absbottom" width="17" height="15"> equals <img border="0" src="../symbols/X1.gif" align="absbottom" width="15" height="15"> + <img border="0" src="../symbols/X2.gif" align="absbottom" width="16" height="15"> + <img border="0" src="../symbols/X3.gif" align="absbottom" width="16" height="15">. The covariance matrix for <i><b>X</b></i> is </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" cellpadding="0" height="36" id="table7"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/multicollinear_[1].gif" width="137" height="105"></td> <td width="5%" align="right">[1]</td> </tr> </table> <p class="body">Random vector <i><b>Z</b></i> is multicollinear. Like <i> <b>X</b></i>, its first three components <img border="0" src="../symbols/Z1.gif" align="absbottom" width="13" height="15">, <img border="0" src="../symbols/Z2.gif" align="absbottom" width="14" height="15"> and <img border="0" src="../symbols/Z3.gif" align="absbottom" width="14" height="15"> are uncorrelated, each with mean 0 and standard deviation 1. The fourth component <img border="0" src="../symbols/Z4.gif" align="absbottom" width="15" height="15"> equals <img border="0" src="../symbols/Z1.gif" align="absbottom" width="13" height="15"> + <img border="0" src="../symbols/Z2.gif" align="absbottom" width="14" height="15"> + <img border="0" src="../symbols/Z3.gif" align="absbottom" width="14" height="15"> + <i>E</i>, where <i>E</i> is a “noise” random variable that is uncorrelated with <img border="0" src="../symbols/Z1.gif" align="absbottom" width="13" height="15">, <img border="0" src="../symbols/Z2.gif" align="absbottom" width="14" height="15"> and <img border="0" src="../symbols/Z3.gif" align="absbottom" width="14" height="15"> and has mean 0 and standard deviation .001. Except for the addition of “noise” <i>E</i>, our random vector <i><b>Z</b></i> is identical to our random vector <i><b>X</b></i>. Its covariance matrix is </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table8"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/multicollinear_[2].gif" width="222" height="77"></td> <td width="5%" align="right">[2]</td> </tr> </table> <p class="body">The covariance matrix of <i><b>X</b></i> is singular. It has determinant 0. The covariance matrix of <i><b>Z</b></i> is not singular, but with a determinant of .000001, it is “almost” singular. The random variable <img border="0" src="../symbols/Z4.gif" align="absbottom" width="15" height="15"> is almost a <a href="../articles/polynomial.htm">linear polynomial</a> of <img border="0" src="../symbols/Z1.gif" align="absbottom" width="13" height="15">, <img border="0" src="../symbols/Z2.gif" align="absbottom" width="14" height="15"> and <img border="0" src="../symbols/Z3.gif" align="absbottom" width="14" height="15">, but not quite. We added just enough random “noise” to make it linearly independent. We say a random vector is multicollinear if it is “almost” singular in this sense. </p> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="165" align="left" id="table9"> <tr> <td width="160"> <script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </td> <td width="11"> </td> </tr> </table> <p class="body">Realizations of a multicollinear random vector tend to cluster near a plane within <img border="0" src="../symbols/real_numers_n_dimensional.gif" align="absbottom" width="18" height="16">. They don’t all lie in that plane, but they “almost” do. This is illustrated with realizations of a two-dimensional multicollinear random vector <i><b>Z</b></i> in Exhibit 1. </p> <div align="center"> <center> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="225" cellpadding="0" id="table10"> <tr> <td> <p class="exhibit_header"><span class="exhibit_header">Realizations of a <br> Multicollinear Random Vector</span><br> <span class="exhibit_subheader">Exhibit 1</span></td> </tr> <tr> <td bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td> <img border="0" src="../images/ex1_Z2.gif" width="150" height="143"></td> </tr> <tr> <td> <p class="exhibit_legend" style="line-height: 150%">Realizations of a multicollinear two-dimensional random vector <i><b>Z</b></i>. Component <i>Z</i><sub>2</sub> is “almost” a linear polynomial of component <i>Z</i><sub>1</sub>.</td> </tr> </table> </center> </div> <p class="body">We may think of a random vector <i><b>Z</b></i> as being “almost” singular if its covariance matrix has a determinant | <img border="0" src="../symbols/sigma_bold_cap_z.gif" align="absbottom" width="19" height="15"> | close to 0. In practical applications, the magnitude of this determinant will depend upon the units in which components of <i><b>Z</b></i> are measured. A more reasonable test for multicollinearity is to consider the determinant of the correlation matrix of <i><b>Z</b></i>. This determinant will always be in the interval [0,1]. If it is very close to 0, this is an indication of multicollinearity. Obviously, if it equals 0, <i><b>Z</b></i> is singular. </p> <p class="body">As describe in the article <i> <a href="../articles/positive_definite_matrix.htm">positive definite matrix</a></i>, the dimensionality of a singular random vector <i><b>X</b></i> can be reduced with a simple change of variables. No information is lost, as we only eliminate extraneous random variables. Multicollinearity is more problematic. Reducing the dimensionality of a multicollinear random vector <i><b>Z</b></i> requires an approximation that somehow identifies and discards minor randomness that is preventing the covariance matrix from being singular. </p>  <p class="body">This is the situation we face with our natural gas portfolio. We feel confident that the natural gas market can reasonably be modeled with less than 564 random variables, but we can’t arbitrarily discard random variables! If our covariance matrix isn’t singular, how can we replace our 564 random variables with a smaller set that convey essentially the same information? <a href="../articles/principal_components.htm">Principal component</a> analysis provides a solution. </p>  <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" id="table11"> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black"><a name="Exercises">Exercises</a></td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table12"> <tr> <td width="100%"> <p class="body" style="margin-left: 10; margin-top: 8; margin-bottom: 8"> <img border="0" src="../images/bullet_exercise.gif" align="absbottom" width="19" height="17">Which of the following covariance matrices <img border="0" src="../symbols/sigma_bold_cap.gif" align="absbottom" width="9" height="15"> are singular? Which are multicollinear?<p style="margin-top: 3; margin-left:10"> <img border="0" src="../formulas/multicolinear_e.gif" width="163" height="321"><p class="body" style="margin-left: 10; margin-top: 8; margin-bottom: 8"> [<a href="../solutions/multicollinear_1.htm">solution</a>]</td> </tr> </table> </td> </tr> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black">Sponsored Links</td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" width="75"> <img border="0" src="../images/ad_side_banner_1.jpg" width="103" height="285"></td> <td bgcolor="#FFFFCC" width="99%"> <script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script>  </td> </tr> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black">Related Internal Links</td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table13"> <tr> <td width="100%"> <p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/cholesky_factorization.htm">Cholesky matrix</a> A lower-triangular matrix that acts as a matrix "square root" for a positive definite matrix.</strong><p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/correlation.htm">correlation</a> A parameter that indicates the tendency for two random variables to "move together" of "co-vary."</strong><p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/eigenvalue.htm">eigenvalue, eigenvector</a> Concepts from linear algebra.</strong></p> <p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/joint_normal_distribution.htm">joint normal distribution</a> A multivariate distribution with normal marginal distributions.<p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/linear_polynomial_of_a_random_vecrtor.htm">linear polynomial of a random vector</a> A random variable or random vector that is defined as a linear polynomial of a random vector.<p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/positive_definite_matrix.htm">positive definite matrix</a> A real symmetric matrix, all of whose eigenvalues are real and positive.<p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/principal_components.htm">principal component analysis</a> A technique for orthogonalizing a random vector.</td> </tr> </table> </td> </tr> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black">Sponsored Links</td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" width="75"> <img border="0" src="../images/ad_side_banner_3.jpg" width="103" height="285"></td> <td bgcolor="#FFFFCC" width="99%">  <p class="ad_legend" style="margin-left: 0; margin-bottom:2"> <a target="_top" href="http://www.riskexpertise.com/link/advertising.htm"> <font color="#000000">Ads by Contingency Analysis</font></a></p> <p style="margin-top: 0; margin-bottom: 0"> <iframe src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=82436;type=iframe" style="margin: 0px; padding: 0px; border-width: 0px" width="336" height="265" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" name="I2"> <![if lt IE 4]> <script src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=82436;type=js" type="text/javascript"> </script> <noscript> <a href="http://servedbyadbutler.com/go2/;ID=147917;size=336x265;setID=82436"> <img src="http://servedbyadbutler.com/adserve.ibs/;ID=147917;size=336x265;setID=82436;type=img" border="0" height="265" width="336" alt=""></a> </noscript><![endif]></iframe>   </td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table14"> <tr> <td width="100%" bgcolor="#FFFFFF"> <p align="center" style="margin-top: 18"><map name="FPMap365"> <area target="_top" href="http://www.riskglossary.com" shape="rect" coords="0, 0, 164, 102"> </map><img border="0" src="../images/logo.jpg" usemap="#FPMap365" width="165" height="103"></p> <p align="center"><span class="verdana10"> <a href="http://www.riskglossary.com/main/disclaimer.htm" target="_self"> Disclaimer</a></span></p> <p align="center">website: <span class="verdana10"> <a target="_top" href="http://www.contingencyanalysis.com"> http://www.contingencyanalysis.com</a></span><br> glossary direct link: <span class="verdana10"> <a target="_top" href="http://www.riskglossary.com"> http://www.riskglossary.com</a></span><br> copyright © Contingency Analysis, 2006<p align="center"><map name="FPMap1"> <area target="_top" href="http://www.contingencyanalysis.com" shape="rect" coords="0, 0, 240, 88"> </map><img border="0" src="../images/logo_glossary_white.gif" usemap="#FPMap1" width="241" height="89"></td> </tr> </table> </td> </tr> </table> </td> <td>  </td> </tr> </table> </body>