The least squares criterion is a statistical approach used to provide the most accurate estimate of relationships between sets of variables in sample data. It is used to define regression lines and planes that yield estimates of a dependent variable, given values for an independent variable.
Least squares analysis is the most popular approach to the computation of regression lines because it is relatively simple and highly accurate. Particularly in linear relationships, it provides the best linear unbiased estimator (BLUE), of sample data. It also gives the maximum likelihood estimator (MLE), in regressions where errors from the regression line form a normal bell-shaped distribution.
In order to understand how a least squares analysis is performed, it is important to first understand the properties of a regression analysis. Simply described, regression analysis uses algebraic formulas to estimate the value of a continuous random dependent variable, using other independent variables as indicators. These formulas produce an estimate of the dependent variable that is most correct (or least incorrect), given any value of an independent variable.
The least squares approach is nearly 100 years older than regression analysis. It was independently developed between 1805 and 1809 by French mathematician Adrien-Marie Legendre (1752-1833) and German mathematician and astronomer Carl Friedrich Gauss (1777-1855). These mathematicians were working on ways to estimate the paths of comets.
Astronomical observations had suggested that these comets maintained highly erratic orbits. In fact, the deviations were due to observational errors. This led the discoverers to develop a method, the least
The mathematicians work yielded important insights into estimating relationships between other types of graphical coordinates—specifically, how to rationalize deviations from a theoretical line of best fit. By the definition of such an estimated line, there are as many negative deviations as positive ones. As a result, all deviations must be expressed as absolute values, a condition met through squaring (the square of any number, whether positive or negative, is positive).
The least squares criterion produces a line in which the sum of the squared deviations of every value Y from the line are lowest. The line represents a continuous estimate of Y values for every value of X, based on the sample data (see Figure 1).
This simple example demonstrates that it is impossible to draw a straight line, or linear regression, that touches all four points on the graph. Instead, we use the least squares criterion to determine the location of a straight line (Q) that comes closest to the points.
Squaring also places the regression line precisely where deviations from the line are lowest. If the sums of deviations were not squared, the line may drift upward or downward until it meets a coordinate. Increased deviation on one side of the line would be
When the deviations are squared, the degree of their deviation is amplified. The difference between 2.5, 3.5, 4.5, and 5.5 is 1, but the difference between their squares—6.25, 12.25, 20.25, 30.25—increases incrementally, 6, 8, and 10.
As a result, the least squares formula indicates a specific slope 13, and places that slope at a specific reference point a, the Y intercept.
[ John Simley ,
updated by Kevin J. Murphy ]
Foster, D. P., R. A. Stine, and R. P. Waterman. Business Analysis Using Regression. New York: Springer-Verlag, 1998.
Lawson, Charles L., and Richard J. Hanson. Solving Least Squares Problems. Society for Industrial and Applied Mathematics, 1995.
Lind, Douglas A., and Robert D. Mason. Basic Statistics for Business and Economics. New York: McGraw-Hill Higher Education, 1996.