|
The method of least
squares is an alternative to interpolation for fitting a function
to a set of points. Unlike interpolation, it does not require the fitted
function to intersect each point. The method of least squares is probably
best known for its use in statistical regression, but it is used in many
contexts unrelated to statistics. The method encompasses many techniques.
We present a fairly general approach called
ordinary least squares.
Suppose researchers gather 10 data points
related to some phenomenon. We
interpolate a ninth-order polynomial based upon the data. See
Exhibits
1 and 2:
|
|
|
 |
|
k |
x[k] |
y[k] |
|
1 |
1.1 |
2.14 |
|
2 |
1.4 |
2.60 |
|
3 |
2.5 |
1.15 |
|
4 |
2.7 |
1.19 |
|
5 |
3.2 |
1.88 |
|
6 |
3.6 |
1.55 |
|
7 |
4.1 |
2.65 |
|
8 |
4.3 |
3.80 |
|
9 |
4.5 |
4.46 |
|
10 |
4.9 |
6.35 |
|
|
Hypothetical data to be used in comparing
interpolation with ordinary least squares. |
|
|
|
|
 |
|
Interpolated ninth-order polynomial. |
Because the polynomial is forced to intercept every point,
it weaves up and down. In some applications, data may reflect random
errors or other sources of “noise.” Forcing a curve to pass through each
point causes its shape to reflect such noise as much as any underlying
process that generated the data. We say the interpolated function is
overfit to the data. As an alternative, we may fit a curve to data without
requiring that it intercept each point. A quadratic polynomial fit in this
manner to the data of Exhibit 1 is illustrated in Exhibit 3.
|
|
|
|
 |
|
A quadratic polynomial fit to the data of
Exhibit 1 using the method of ordinary least squares. |
The polynomial of Exhibit 3 was constructed with the method of ordinary
least squares. The form of the polynomial was specified as follows
|
 |
[1] |
with the constants
 determined in such a manner as to
minimize the sum of squares
|
 |
[2] |
Consider l points
 where
 , and
 . We wish
to fit a function
 of form
|
 |
[3] |
to the data in such a manner as to minimize the sum of
squares
|
 |
[4] |
As with ordinary interpolation, functions
 :
can take any form. Unlike interpolation, we require
that the number m of functions be less than the number l of points.
Let’s express our problem with matrices. Define
|
 |
[5] |
This is unknown. It is what we want to solve for. Define
f as the l
 m
matrix comprising values of each function
evaluated at each point
:
|
 |
[6] |
Define the vector
|
 |
[7] |
Both the matrix f and vector y are constants. They are known. We
express sum-of-squares formula [4] as
|
 |
[8] |
| [9] |
This is a quadratic polynomial in
 . It can
be shown that, if f has linearly independent columns, it has
a unique minimum, which occurs at
|
 |
[10] |
Continuing with our example of fitting a quadratic polynomial to the
data of Exhibit 1, we seek a polynomial
|
 |
[11] |
where
 f1(x) = 1,
f2(x) =
x,
f3(x) =
x2.
Let
|
 |
[12] |
We have
|
 |
[13] |
and
|
 |
[14] |
Applying [10], we obtain
|
 |
[15] |
and our quadratic polynomial is
|
 |
[16] |
which was graphed in Exhibit 3.
|
|
|
|
|
 Use ordinary least squares to fit a linear
polynomial
|
 |
[e1] |
to the five points indicated in Exhibit
E1.
[solution]
Use ordinary least squares to fit a linear
polynomial
|
 |
[e2] |
to the five points indicated in Exhibit
E2.
[solution]
Prove that, if the number m of
functions
equals the number l
of points
,
then the least squares solution [10] reduces to the
interpolation solution [Interpolation
15]. In this regard, ordinary least
squares is a generalization of ordinary interpolation. [solution]
|
|
|
|
|
|
 |
Ads by Contingency Analysis
|
|
|
|
|
|
 cubic spline interpolation Any of several methods of interpolating with
cubic splines.
interpolation Any procedure for fitting a function to a set of points in
such a manner that the function intercepts each of the points.
remapping
In value-at-risk, the approximation of one risk vector with another.
linear
and quadratic polynomials Two basic forms of polynomials.
Taylor
series expansion In calculus, a power series obtained as a
limit of Taylor polynomials that may approximate or equal the
function from which it is constructed. |
|
|
|
|
|
 |
Ads by Contingency Analysis
|
|
|
|
|
