Principal Component Analysis

<body onLoad="MM_checkPlugin('Shockwave Flash','','http://www.riskglossary.com/default_nofl.htm',false);return document.MM_returnValue" stylesrc="../_private/template.htm" bgcolor="#FFFFFF" link="#669933" vlink="#000000" alink="#000000"> <div align="left"> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" id="table1"> <tr> <td width="740"> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" height="37" id="table2"> <tr> <td height="35" width="100%"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" id="table3"> <tr> <td width="100%" align="left" valign="top"> <p class="header_gold">Principal Component Analysis</td> <td width="45" align="right" valign="bottom"> <map name="FPMap6"> <area href="http://www.contingencyanalysis.com" shape="rect" coords="1, 0, 44, 11" target="_top"> </map> <img border="0" src="../images/home.gif" usemap="#FPMap6" width="45" height="12"></td> </tr> </table> </td> </tr> <tr> <td width="100%" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td width="0%" valign="top" bgcolor="#FFFFCC"> <table border="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" cellspacing="1" id="table4"> <tr> <td width="0%" align="left" valign="top"> <p class="header2_black">Explained:<br> <img border="0" src="../images/blank_1x100.gif" width="100" height="1"></td> <td width="95%" align="left" valign="top"><p class="word_list"> <a href="#principal components">principal component</a><p class="word_list"> <a href="#principal component analysis">principal component analysis</a><br> <img border="0" src="../images/blank_1x160.gif" width="160" height="1"></td> </tr> </table> </td> </tr> </table> <table border="0" cellpadding="0" style="border-collapse: collapse" bordercolor="#111111" width="310" align="right" id="table5"> <tr> <td align="right" width="9" height="14"></td> <td width="300" height="14"> <font size="2"> </font><font size="4"> </font></td> </tr> <tr> <td align="right" width="9"> </td> <td width="300"> <script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script>  </td> </tr> </table> <p class="body">With <b><a name="principal component analysis">principal component analysis</a></b>, we transform a random vector <i><b>Z</b></i> with <a href="../articles/correlation.htm">correlated</a> components <img border="0" src="../symbols/Zi.gif" align="absbottom" width="12" height="15"> into a random vector <i><b>D</b></i> with uncorrelated components <img border="0" src="../formulas/principal_components_Di.gif" align="absbottom" width="14" height="15">. This is called an orthogonalization of <i><b>Z</b></i>. </p> <p class="body">Principal component analysis can be performed on any random vector <i><b>Z</b></i> whose second moments exist, but it is most useful with <a href="../articles/multicollinear.htm">multicollinear</a> random vectors. Principal component analysis takes the hyperplane in which realizations of a multicollinear random vector “almost” sit and aligns it with the coordinate system of <img border="0" src="../symbols/real_numers_n_dimensional.gif" align="absbottom" width="18" height="16">. The components of <i><b>D</b></i> that are perpendicular to the transformed plane have small, almost trivial <a href="../articles/standard_deviation.htm">standard deviations</a>. Discarding these components provides a lower-dimensional approximate representation for <i><b>Z</b></i>. This is illustrated with realizations of a multicollinear two-dimensional random vector <i><b>Z</b></i> in Exhibit 1: </p> <div align="center"> <center> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="350" cellpadding="0" id="table6"> <tr> <td> <p class="exhibit_header"><span class="exhibit_header">Dimension Reduction with Principal Component Analysis</span><br> <span class="exhibit_subheader">Exhibit 1</span></td> </tr> <tr> <td bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td> <p style="margin-top: 10"> <img border="0" src="../images/ex1_principal_components.gif" width="533" height="143"></td> </tr> <tr> <td> <p class="exhibit_legend">Principal component analysis can be used to reduce the dimensionality of a multicollinear random vector. Realizations for a multicollinear two-dimensional random vector <i> <b>Z</b></i> are illustrated in the left graph. Principal component analysis transforms <i><b>Z</b></i> into an equivalent multicollinear random vector <i><b>D</b></i> that is aligned with the coordinate system. Realizations of <i><b>D</b></i> are shown in the middle graph. Discarding the second component <img border="0" src="../formulas/principal_components_d2_exhibit_text.gif" align="absbottom" width="14" height="13"> transforms <i><b>D</b></i> into a one-dimensional approximate representation of the two-dimensional <i><b>Z</b></i>. Realizations of this representation are shown in the right graph.</td> </tr> </table> </center> </div>  <p class="body"><span class="header_section_green">Example: European Currencies </span><br> Let's start with an intuitive example before presenting the formal mathematics. Suppose today is June 30, 2000. We consider a random vector <i><b>Z</b></i> whose components represent the simple price returns that specific European currencies will realize versus the US dollar (USD) over the upcoming trading day: </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table7"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[1].gif" width="333" height="173"></td> <td width="5%" align="right">[1]</td> </tr> </table> <p class="body">Exhibit 2 graphs 18 months of daily exchange-rate data drawn from the period immediately following the launch of the new euro (<a href="../articles/currency_codes.htm">EUR</a>) currency. In our data, the EUR weakens following its launch, and the remaining European currencies—those that did not join the EUR on January 1, 1999—weaken in sympathy. All the currencies track the EUR, but the <a href="../articles/currency_codes.htm">GBP</a> does so the least. It is less correlated with the EUR and loses value more slowly. </p> <div align="center"> <center> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="350" cellpadding="0" id="table8"> <tr> <td> <p class="exhibit_header"><span class="exhibit_header">European Exchange Rates: <br> January 1999 Through June 2000 </span><br> <span class="exhibit_subheader">Exhibit 2</span></td> </tr> <tr> <td bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td> <p style="margin-top: 10"> <img border="0" src="../images/ex2_principal_components.gif" width="400" height="274"></td> </tr> <tr> <td> <p class="exhibit_legend">Historical exchange rates versus the <a href="../articles/currency_codes.htm">USD</a> for the period January 1, 1999, through June 30, 2000. Exchange rates are presented as USD/unit of currency, so a rising curve indicates a strengthening currency. Exchange rates are individually scaled so they all fit on the graph. </td> </tr> </table> </center> </div> <p class="body">We assume <img border="0" src="../formulas/principal_components_mu_z.gif" align="absbottom" width="18" height="12"> = <b>0</b>. Based upon a time series analysis of the historical price data, we construct a covariance matrix for <i><b><a name="[2]">Z</a></b></i> </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table9"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[2].gif" width="507" height="120"></td> <td width="5%" align="right">[2]</td> </tr> </table> <p class="body">The corresponding correlation matrix is: </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table10"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_component_[3].gif" width="358" height="126"></td> <td width="5%" align="right">[3]</td> </tr> </table> <p class="body">The correlations are all positive. Several exceed 0.90. The one between <a href="../articles/currency_codes.htm">DKK</a> and EUR exceeds 0.99. The smallest is a respectable 0.45 between GBP and <a href="../articles/currency_codes.htm">SEK</a>. With such pronounced interdependencies between its components, we expect <i><b>Z</b></i> to be <a href="../articles/multicollinear.htm">multicollinear</a>, and it is. The correlation matrix has determinant |<img border="0" src="../formulas/principal_components_correlation_matrix_z.gif" align="absbottom" width="17" height="12">| = .0000045. </p> <p class="body">To define <b><a name="principal components">principal components</a></b> of <i><b>Z</b></i>, we calculate orthonormal (orthogonal and of unit length) <a href="../articles/eigenvalue.htm"> eigenvectors</a> <img border="0" src="../formulas/principal_components_vi.gif" align="absbottom" width="10" height="11"> of the covariance matrix <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15"> of <i><b>Z</b></i>. We arrange these as the columns of a matrix: </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table11"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[4].gif" width="390" height="156"></td> <td width="5%" align="right">[4]</td> </tr> </table> <p class="body">The eigenvectors <img border="0" src="../formulas/principal_components_vi.gif" align="absbottom" width="10" height="11"> are graphed in Exhibit 3. Corresponding <a href="../articles/eigenvalue.htm">eigenvalues</a> <img border="0" src="../symbols/lambda_i.gif" align="absbottom" width="12" height="16"> are also indicated: </p> <div align="center"> <center> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="450" cellpadding="0" id="table12"> <tr> <td> <p class="exhibit_header"><span class="exhibit_header">Example: Eigenvectors for Currency Returns</span><br> <span class="exhibit_subheader">Exhibit 3</span></td> </tr> <tr> <td bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td> <p style="margin-top: 8"> <img border="0" src="../images/ex3_principal_components.gif" width="450" height="623"></td> </tr> <tr> <td> <p class="exhibit_legend">Eigenvectors of covariance matrix [<a href="#[2]">2</a>]. Corresponding eigenvalues are also indicated.</td> </tr> </table> </center> </div> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="345" align="right" id="table13"> <tr> <td align="right" width="9" height="14"></td> <td width="336" height="14"> <font size="2"> </font><font size="4"> </font></td> </tr> <tr> <td align="right" width="9"> </td> <td width="336"> <script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script>  </td> </tr> </table> <p class="body">The eigenvectors may be thought of as “modes of fluctuation” of random vector <i><b>Z</b></i>. We observed in our historical data a tendency for the European currencies to move together. This is reflected in the first eigenvector. It describes a broad move in all the currencies, with the GBP participating about half as much as the other currencies. The second eigenvector has the GBP moving in opposition to the <a href="../articles/currency_codes.htm">NOK</a> and SEK, with the <a href="../articles/currency_codes.htm">CHF</a> moving modestly with the GBP. The third eigenvector describes the GBP, NOK, and SEK moving together in opposition to the other currencies. The remaining eigenvectors describe other “modes of fluctuation.” </p> <p class="body">If the eigenvectors <img border="0" src="../formulas/principal_components_vi.gif" align="absbottom" width="10" height="11"> are modes of fluctuations of <i><b>Z</b></i>, then <i><b>Z</b></i> is a random combination of those modes of <a name="[5]">fluctuation</a>: </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" cellpadding="0" height="36" id="table14"> <tr> <td width="100%" align="center"> <img border="0" src="../images/principal_components_[5].gif" width="217" height="26"></td> <td width="5%" align="right">[5]</td> </tr> </table> <p class="body">The <img border="0" src="../formulas/principal_components_Di.gif" align="absbottom" width="14" height="15"> are the principal components of Z. They are random variables that define each mode of fluctuation’s random contribution to Z. The <img border="0" src="../formulas/principal_components_Di.gif" align="absbottom" width="14" height="15"> are uncorrelated with <a href="../articles/standard_deviation.htm"> variances</a> equal to the eigenvalues of their corresponding eigenvectors. The vector <i><b>D</b></i> of principal components has mean <img border="0" src="../formulas/principal_components_mu_D.gif" align="absbottom" width="18" height="12"> = <b>0 </b>and covariance <a name="[6]">matrix</a> </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table15"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[6].gif" width="517" height="91"></td> <td width="5%" align="right">[6]</td> </tr> </table> <p class="body">We have ordered our principal components according to their variances. From our covariance matrix <img border="0" src="../formulas/principal_components_sigma_D.gif" align="absbottom" width="19" height="15">, we see that the first three principal components are more significant than the rest. The last two principal components, <img border="0" src="../formulas/principal_components_D_D.gif" align="absbottom" width="64" height="15">, have variances that are less than 1% of the variance of <img border="0" src="../formulas/principal_components+D1.gif" align="absbottom" width="15" height="15">. Their contribution to random vector <i><b>Z</b></i> is trivial. </p> <p class="body">We can approximate <i><b>Z</b></i> by discarding from [<a href="#[5]">5</a>] insignificant principal components. The more we discard, the simpler—and cruder!—will be our approximation. If we want to be aggressive in our approximation, we can discard the contributions of the last four principal components, and approximate <i><b>Z</b></i> with just the first three. A more accurate approximation can be obtained by discarding only the last two. For this example, we pursue the more aggressive course. We <a name="[7]">define</a> </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table16"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[7].gif" width="147" height="25"></td> <td width="5%" align="right">[7]</td> </tr> </table> <p class="body">and approximate <i><b>Z</b></i> with <img border="0" src="../formulas/principal_components_z_tilde.gif" align="absbottom" width="11" height="19">. Like <i><b>Z</b></i>, <img border="0" src="../formulas/principal_components_z_tilde.gif" align="absbottom" width="11" height="19"> has mean vector <b>0</b>. Its covariance matrix is obtained from [<a href="#[6]">6</a>] and [<a href="#[7]">7</a>] (see the article <i> <a href="../articles/linear_polynomial_of_a_random_vecrtor.htm">linear polynomial of a random vector</a></i>): </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table17"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[8].gif" width="506" height="117"></td> <td width="5%" align="right">[8]</td> </tr> </table> <p class="body">Comparing this covariance matrix with [<a href="#[2]">2</a>], you can judge for yourself the quality of the approximation. </p>  <p class="body"><span class="header_section_green">Principal Components </span><br> Our example informally introduced principal components. Now let’s formalize them. Consider an <i>n</i>-dimensional random vector <i><b>Z</b></i> with mean <img border="0" src="../formulas/principal_components_mu_z.gif" align="absbottom" width="18" height="12"> and nonsingular covariance matrix <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15">. We construct principal components in such a manner that the first accounts for as much of the variability of <i><b>Z</b></i> as possible. The second accounts for as much of the remaining variability of <i><b>Z</b></i> as possible, and so on. </p> <p class="body">Specifically, the first principal component <img border="0" src="../formulas/principal_components+D1.gif" align="absbottom" width="15" height="15"> is defined as </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table18"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[9].gif" width="102" height="25"></td> <td width="5%" align="right">[9]</td> </tr> </table> <p class="body">where <img border="0" src="../formulas/principal_components_v1.gif" align="absbottom" width="10" height="11"> has unit length and is selected to maximize the variance of <img border="0" src="../formulas/principal_components+D1.gif" align="absbottom" width="15" height="15">. This is achieved by setting <img border="0" src="../formulas/principal_components_v1.gif" align="absbottom" width="10" height="11"> equal to the normalized first eigenvector of <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15"> – the eigenvector with the largest eigenvalue. In this case, the variance of <img border="0" src="../formulas/principal_components+D1.gif" align="absbottom" width="15" height="15"> equals that eigenvalue, <img border="0" src="../symbols/lambda_1.gif" align="absbottom" width="12" height="16">. </p> <p class="body">The second principal component <img border="0" src="../formulas/principal_components_D2.gif" align="absbottom" width="18" height="15"> is defined as </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table19"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[10].gif" width="102" height="25"></td> <td width="5%" align="right">[10]</td> </tr> </table> <p class="body">where <img border="0" src="../formulas/principal_components_v2.gif" align="absbottom" width="12" height="11"> is selected from the set of all <i>n</i>-dimensional unit vectors that are orthogonal to <img border="0" src="../formulas/principal_components_v1.gif" align="absbottom" width="10" height="11"> in such a manner as to maximize the variance of <img border="0" src="../formulas/principal_components_D2.gif" align="absbottom" width="18" height="15">. This is achieved by setting <img border="0" src="../formulas/principal_components_v2.gif" align="absbottom" width="12" height="11"> equal to the normalized second eigenvector of <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15"> – the eigenvector with the second largest eigenvalue. The variance of <img border="0" src="../formulas/principal_components_D2.gif" align="absbottom" width="18" height="15"> equals that eigenvalue, <img border="0" src="../symbols/lambda_2.gif" align="absbottom" width="14" height="16">. </p> <p class="body">Proceeding in this manner, we define the remaining principal components. There will be <i>m</i> principal components <img border="0" src="../formulas/principal_components_Di.gif" align="absbottom" width="14" height="15">, each one corresponding to a normalized eigenvector <img border="0" src="../formulas/principal_components_vi.gif" align="absbottom" width="10" height="11"> of <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15">. We can <a name="[11]">represent </a></p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table20"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[11].gif" width="287" height="26"></td> <td width="5%" align="right">[11]</td> </tr> </table> <p class="body">The vector of principal components <i><b>D</b></i> has mean <img border="0" src="../formulas/principal_components_mu_D.gif" align="absbottom" width="18" height="12"> = <b>0</b> and covariance matrix </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="0" height="36" id="table21"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[12].gif" width="166" height="76"></td> <td width="5%" align="right">[12]</td> </tr> </table> <p class="body">If <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15"> is nonsingular, the number <i>m</i> of principal components equals the dimensionality <i>n</i> of <i><b>Z</b></i>. If <img border="0" src="../formulas/principal_components_sigma_z.gif" align="absbottom" width="19" height="15"> is singular, some of its eigenvalues will equal 0, and the number <i>m</i> of principal components will be less than the dimensionality <i>n</i> of <i><b>Z</b></i>. In this case, [12] will have reduced the dimensionality of the singular <i><b>Z</b></i> in exactly the same manner as that described in the article <i> <a href="../articles/positive_definite_matrix.htm">positive definite, positive semidefinite covariance matrix</a></i>. </p> <p class="body"><span class="header_section_green">Choice of Weights</span><br> Principal component analysis is best performed on random variables whose standard deviations are reflective of their relative significance for an application. This is because principal component analysis depends upon both the correlations between random variables and the standard deviations of those random variables. If we were to change standard deviations of a set of random variables but leave their correlations the same, this would change their principal components. In a sense, principal component analysis uses standard deviation as a <a href="../articles/risk_metric_and_risk_measure.htm">metric</a> of significance. If one random variable has a standard deviation that far exceeds the rest, that random variable will dominate the first eigenvector. </p> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="345" align="left" id="table22"> <tr> <td width="168" height="15"> <p class="ad_legend"> <a target="_top" href="http://www.riskexpertise.com/link/advertising.htm"> <font color="#000000">Ads by Contingency Analysis</font></a></td> <td width="168" height="15" align="right"> <p class="ad_legend"> <a target="_top" href="http://www.riskexpertise.com/link/advertising.htm"> <font color="#000000">Advertise on this site</font></a></td> <td width="9" rowspan="2"> </td> </tr> <tr> <td width="336" colspan="2">  <iframe src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=84506;type=iframe" allowtransparency="true" background-color="transparent" style="margin: 0px; padding: 0px; border-width: 0px" width="336" height="265" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" name="I1"> <![if lt IE 4]> <script src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=84506;type=js" type="text/javascript"> </script> <noscript> <a href="http://servedbyadbutler.com/go2/;ID=147917;size=336x265;setID=84506"> <img src="http://servedbyadbutler.com/adserve.ibs/;ID=147917;size=336x265;setID=84506;type=img" border="0" height="265" width="336" alt=""></a> </noscript><![endif]></iframe>  </td> </tr> </table> <p class="body">Unfortunately, there may be no correspondence between a random variable’s standard deviation and its significance. Standard deviations depend upon the units in which a random variable is measured. Suppose a random variable reflects the time it takes for some event to occur, and if the random variable is measured in days, it has a standard deviation of 13.5. If the standard deviation is measured in hours, it is 324. Measured in minutes, it becomes 19,440. Certainly, the 19,440 standard deviation is no more significant than the 13.5 standard deviation, but principal component analysis will treat it as more significant! </p> <p class="body">If we use principal components only to orthogonalize a random vector, this will not be a problem. No information is lost. It will be a problem if principal components are discarded to form an approximation. In this case, information is lost. Before we discard principal components that appear “insignificant,” we should make sure that they truly are insignificant. </p> <p class="body">There are various solutions to this problem. We might insist that all random variables be measured in the same units, but this is not always feasible. If one random variable represents temperature and another represents volume, these are fundamentally different quantities. Also, identical units do not necessarily correspond to identical significance. Suppose we are analyzing blood samples for lead, and we have a random variable for each component of the blood. All components are measured in parts per million (ppm). Measured in ppm, the standard deviation of lead will be trivial compared to standard deviations for other constituents of the blood. Yet, the lead component is the most important random variable! </p> <p class="body">Alternatively, we might apply principal component analysis to normalized random variables obtained by dividing each random variable by its standard deviation: </p> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" cellpadding="0" height="36" id="table23"> <tr> <td width="100%" align="center"> <img border="0" src="../formulas/principal_components_[13].gif" width="56" height="50"></td> <td width="5%" align="right">[13]</td> </tr> </table> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="345" align="right" id="table24"> <tr> <td align="right" width="9" height="14"></td> <td width="336" height="14"> <font size="2"> </font><font size="4"> </font></td> </tr> <tr> <td align="right" width="9"> </td> <td width="336"> <script type="text/javascript"></script> <script type="text/javascript" src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script>  </td> </tr> </table> <p class="body">With this approach, we effectively apply principal component analysis to the random variables’ correlation matrix. This represents a different weighting from that obtained by measuring all random variables in identical units—but not necessarily a better one. </p> <p class="body" style="margin-bottom: 36">Any solution may be reasonable in certain contexts and unreasonable in others. Each one weights the random variables in some manner. There is no objective way to assign weights, just as there is no objective way to assign “significance.” Weights and “significance” can and should vary from one application to another. When we use principal components to reduce the dimensionality of a random vector, there is subjectivity in the process. </p> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" id="table25"> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black"><a name="Exercises">Exercises</a></td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table26"> <tr> <td width="100%"> <p class="body" style="margin-left: 10; margin-top: 8; margin-bottom: 8"> <img border="0" src="../images/bullet_exercise.gif" align="absbottom" width="19" height="17">Consider a random vector <i><b>Z</b></i> with mean and covariance matrix<table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" id="table27"> <tr> <td width="98%" align="center"> <p class="body"> <img border="0" src="../formulas/principal_components_ex03_21_69.gif" width="311" height="80"></td> <td width="2%"> <p class="body" align="right">[e1]</td> </tr> </table> <p style="margin-left: 10">a. Calculate the determinant of the corresponding correlation matrix. </p> <p style="margin-left: 10">b. Is <i><b>Z</b></i> singular, multicollinear, or neither of these? </p> <p style="margin-left: 10">c. Calculate the eigenvalues and eigenvectors of <img border="0" src="../formulas/principal_components_ex03_21_sigma_z.gif" align="absbottom" width="22" height="15">.</p> <p style="margin-left: 10">d. Represent <i><b>Z</b></i> in terms of its principal components as in [<a href="#[11]">11</a>]. </p> <p style="margin-left: 10">e. What is the covariance matrix <img border="0" src="../formulas/principal_components_ex03_21_sigma_d.gif" align="absbottom" width="22" height="15"> of the vector of principal components <i><b>D</b></i>? </p> <p style="margin-left: 10">f. Construct an approximation <img border="0" src="../formulas/principal_components_ex03_21_z_tilde.gif" align="absbottom" width="11" height="18"> for <i><b>Z</b></i> based on the first two principal components of <i><b>Z</b></i>. </p> <p style="margin-left: 10">g. Construct the covariance matrix of <img border="0" src="../formulas/principal_components_ex03_21_z_tilde.gif" align="absbottom" width="11" height="18">. Compare your result with the covariance matrix of <i><b>Z</b></i>.<p class="body" style="margin-left: 10; margin-top: 8; margin-bottom: 8"> [<a href="../solutions/principal_components_1.htm">solution</a>]</td> </tr> </table> </td> </tr> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black">Related Internal Links</td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table28"> <tr> <td width="100%"> <p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/cholesky_factorization.htm">Cholesky matrix</a> A lower-triangular matrix that acts as a matrix "square root" for a positive definite matrix.</strong><p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/correlation.htm">correlation</a> A parameter that indicates the tendency for two random variables to "move together" of "co-vary."</strong><p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/eigenvalue.htm">eigenvalue, eigenvector</a> Concepts from linear algebra.</strong></p> <p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/joint_normal_distribution.htm">joint normal distribution</a> A multivariate distribution with normal marginal distributions.<p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/linear_polynomial_of_a_random_vecrtor.htm">linear polynomial of a random vector</a> A random variable or random vector that is defined as a linear polynomial of a random vector.<p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/multicollinear.htm">multicollinear</a> A covariance matrix is muticollinear if it is "almost" singular.</strong><p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><a href="../articles/positive_definite_matrix.htm">positive definite matrix</a> A real symmetric matrix, all of whose eigenvalues are real and positive.<p class="explanatory_text"> <img border="0" src="../images/bullet_link_glossary.gif" width="26" height="15"><strong style="font-weight: 400"><a href="../articles/remappings.htm">remapping</a> In value-at-risk, the approximation of one risk vector with another.</strong></td> </tr> </table> </td> </tr> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black">Sponsored Links</td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" width="75"> <img border="0" src="../images/ad_side_banner_3.jpg" width="103" height="285"></td> <td bgcolor="#FFFFCC" width="99%">  <p class="ad_legend" style="margin-left: 0; margin-bottom:2"> <a target="_top" href="http://www.riskexpertise.com/link/advertising.htm"> <font color="#000000">Ads by Contingency Analysis</font></a></p> <p style="margin-top: 0; margin-bottom: 0"> <iframe src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=82436;type=iframe" style="margin: 0px; padding: 0px; border-width: 0px" width="336" height="265" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" name="I2"> <![if lt IE 4]> <script src="http://servedbyadbutler.com/adserve/;ID=147917;size=336x265;setID=82436;type=js" type="text/javascript"> </script> <noscript> <a href="http://servedbyadbutler.com/go2/;ID=147917;size=336x265;setID=82436"> <img src="http://servedbyadbutler.com/adserve.ibs/;ID=147917;size=336x265;setID=82436;type=img" border="0" height="265" width="336" alt=""></a> </noscript><![endif]></iframe>   </td> </tr> <tr> <td height="30" valign="bottom" colspan="2"> <p class="header2_black">Related Forum Discussions</td> </tr> <tr> <td colspan="2" bgcolor="#CC9966" height="2"> <img border="0" src="../images/blank_2x2.gif" width="2" height="2"></td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table29"> <tr> <td width="100%"> <p class="explanatory_text"> <img border="0" src="../images/bullet_bullhorn_narrow.gif" align="left" width="27" height="22"><b><a target="_top" href="http://www.riskarchive.com/link_thread/03-3/00000222.htm">PCA with correlation matrix</a></b> <i>06 Aug 2003</i><br> Principal components of the yield curve.</td> </tr> </table> </td> </tr> <tr> <td bgcolor="#FFFFCC" colspan="2"> <table border="0" cellspacing="0" style="border-collapse: collapse" bordercolor="#111111" width="100%" cellpadding="6" id="table30"> <tr> <td width="100%" bgcolor="#FFFFFF"> <p align="center" style="margin-top: 18"><map name="FPMap365"> <area target="_top" href="http://www.riskglossary.com" shape="rect" coords="0, 0, 164, 102"> </map><img border="0" src="../images/logo.jpg" usemap="#FPMap365" width="165" height="103"></p> <p align="center"><span class="verdana10"> <a href="http://www.riskglossary.com/main/disclaimer.htm" target="_self"> Disclaimer</a></span></p> <p align="center">website: <span class="verdana10"> <a target="_top" href="http://www.contingencyanalysis.com"> http://www.contingencyanalysis.com</a></span><br> glossary direct link: <span class="verdana10"> <a target="_top" href="http://www.riskglossary.com"> http://www.riskglossary.com</a></span><br> copyright © Contingency Analysis, 2006<p align="center"><map name="FPMap1"> <area target="_top" href="http://www.contingencyanalysis.com" shape="rect" coords="0, 0, 240, 88"> </map><img border="0" src="../images/logo_glossary_white.gif" usemap="#FPMap1" width="241" height="89"></td> </tr> </table> </td> </tr> </table> </td> <td>  </td> </tr> </table> </div> </body>