| View previous topic :: View next topic |
| Author |
Message |
Lurfys Maw
Guest
|
 Posted: Thu Jul 31, 2008 7:09 am Post subject: Why is it "least squares" and not "least absolute diffrerenc |
 |
|
Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those? |
|
| |
|
 Back to top |
Peter Spellucci
Guest
|
| Posted: Thu Jul 31, 2008 10:22 am Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
In article <jjk2949aog5ag5517njt6vuujqedr3jc4b@4ax.com>,
Lurfys Maw <invalid@invalid.invalid> writes:
[quote]Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those?
[/quote]
both methods are in use:
in the sense of statistics the least squares method requires the
assumption of identically normally distributed random errors in the
data (not the regressors) with zero mean.
the gauss-markov theorem then states thta the expectation values of the
computed parameters are the "true" ones.
In practice one seldom bothers about these assumptions but takes
this approach because it leads to a differentiable optimization
problem which can be solved efficiently.
The least squares method however is quite sensitive to outliers in the
data.
for "robust" regression the sum of absolute deviations is used, but this is
originally a nonsmooth problem. transformation to a smooth
problem results in a large constraint optimization problem which to solve
is much harder.
hth
peter |
|
| |
|
| Back to top |
Guest
|
| Posted: Thu Jul 31, 2008 11:45 am Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
On 31 jul, 12:22, spellu...@fb04373.mathematik.tu-darmstadt.de (Peter
Spellucci) wrote:
[quote]In article <jjk2949aog5ag5517njt6vuujqedr3j...@4ax.com>,
Lurfys Maw <inva...@invalid.invalid> writes:
>Why is the "least squares" method used in calculating a regression
>value rather than a "least absolute differences"?
>Why are the differences squared? Why not take the absolute values of
>the differences and sum those?
both methods are in use:
in the sense of statistics the least squares method requires the
assumption of identically normally distributed random errors in the
data (not the regressors) with zero mean.
the gauss-markov theorem then states thta the expectation values of the
computed parameters are the "true" ones.
In practice one seldom bothers about these assumptions but takes
this approach because it leads to a differentiable optimization
problem which can be solved efficiently.
The least squares method however is quite sensitive to outliers in the
data.
for "robust" regression the sum of absolute deviations is used, but this is
originally a nonsmooth problem. transformation to a smooth
problem results in a large constraint optimization problem which to solve
is much harder.
[/quote]
Well said. More info at:
http://hdebruijn.soo.dto.tudelft.nl/jaar2008/index.htm
Han de Bruijn |
|
| |
|
| Back to top |
Dann Corbit
Guest
|
| Posted: Thu Jul 31, 2008 12:26 pm Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
"Lurfys Maw" <invalid@invalid.invalid> wrote in message
news:jjk2949aog5ag5517njt6vuujqedr3jc4b@4ax.com...
[quote]Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those?
[/quote]
Look at the first paragraph of this:
http://en.wikipedia.org/wiki/Least_squares
** Posted from http://www.teranews.com ** |
|
| |
|
| Back to top |
Guest
|
| Posted: Thu Jul 31, 2008 6:33 pm Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
On Jul 31, 12:54 am, Lurfys Maw <inva...@invalid.invalid> wrote:
[quote]Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those?
[/quote]
The reason least squares is used more is historical. It has a simple
derivation for the values of the parameters that minimize the error
sum of squares based on differential calculus. This is how Gauss
discovered it many years ago. Simply put, you can solve for the
regression paramters that minimize the sum of squared residuals using
simple calculus.
Also, the objective function can also be put into the exponent of the
normal distribution. From there, you can show that the least squares
estimate of the regression parameters is also the maximum likelihood
estimate for the normal distribution. From that, you can derive
distributional results for the best regression coefficients.
Basically, least squares is associated with a lot of nice analytical
results. Least absolute differences does not have a simple analytical
solution like least squares. Also, the distribution of the least
absolute difference solution to the "regression" equations is not as
easy.
Mark |
|
| |
|
| Back to top |
Gordon Sande
Guest
|
| Posted: Thu Jul 31, 2008 7:04 pm Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
On 2008-07-31 08:45:55 -0300, umumenu@gmail.com said:
[quote]On 31 jul, 12:22, spellu...@fb04373.mathematik.tu-darmstadt.de (Peter
Spellucci) wrote:
In article <jjk2949aog5ag5517njt6vuujqedr3j...@4ax.com>,
Lurfys Maw <inva...@invalid.invalid> writes:
>Why is the "least squares" method used in calculating a regression
>value rather than a "least absolute differences"?
>Why are the differences squared? Why not take the absolute values of
>the differences and sum those?
both methods are in use:
in the sense of statistics the least squares method requires the
assumption of identically normally distributed random errors in the
data (not the regressors) with zero mean.
the gauss-markov theorem then states thta the expectation values of th
e
computed parameters are the "true" ones.
In practice one seldom bothers about these assumptions but takes
this approach because it leads to a differentiable optimization
problem which can be solved efficiently.
The least squares method however is quite sensitive to outliers in the
data.
for "robust" regression the sum of absolute deviations is used, but th
is is
originally a nonsmooth problem. transformation to a smooth
problem results in a large constraint optimization problem which to so
lve
is much harder.
Well said. More info at:
http://hdebruijn.soo.dto.tudelft.nl/jaar2008/index.htm
Han de Bruijn
[/quote]
Least Absolute Deviations is what you get if you assume Laplacian,
often called double exponential, errors. The estimates are no
longer simply derived using introductory calculus, but rather
requires mathematical programming. LAD regression is equivalent
to linear programming. Robust regression to deal with even longer
tailed distributions than Laplacian requires more elaborate methods.
In the one dimensional case the assumption of Gaussian errors leads to
means and the assumption of Laplacian errors leads to medians.
One view is expressed in "The Gaussian hare and the Laplacian tortoise:
computability of squared-error versus absolute-error estimators".
There is an old saying the physicists believe that Mathematics has proven
that errors are Gaussian while mathematicians believe that Physics has
demonstrated that errors are Gaussian. Neither is true but the assumption
makes things much easier for both groups. There is much current discussion
that finance should be based on errors which are even longer tailed than
Laplacian. |
|
| |
|
| Back to top |
Lurfys Maw
Guest
|
| Posted: Thu Jul 31, 2008 7:05 pm Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
On Thu, 31 Jul 2008 12:22:42 +0200 (CEST),
spellucci@fb04373.mathematik.tu-darmstadt.de (Peter Spellucci) wrote:
[quote]
In article <jjk2949aog5ag5517njt6vuujqedr3jc4b@4ax.com>,
Lurfys Maw <invalid@invalid.invalid> writes:
Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those?
both methods are in use:
in the sense of statistics the least squares method requires the
assumption of identically normally distributed random errors in the
data (not the regressors) with zero mean.
the gauss-markov theorem then states thta the expectation values of the
computed parameters are the "true" ones.
In practice one seldom bothers about these assumptions but takes
this approach because it leads to a differentiable optimization
problem which can be solved efficiently.
The least squares method however is quite sensitive to outliers in the
data.
for "robust" regression the sum of absolute deviations is used, but this is
originally a nonsmooth problem. transformation to a smooth
problem results in a large constraint optimization problem which to solve
is much harder.
[/quote]
Whew. Is there any way to put that in "lay" terms?
What>s special about the power of "2" vs "1" or "3" or "1.5" or "pi"
or "e" or ...?
Why not least cubes? That would be even more sensitive to outliers,
no? Why is squares just the right amount of sensitive, whereas "3" is
too sensitive and "1" not enough?
You probably answered this, but I can>t grasp it. |
|
| |
|
| Back to top |
Gordon Sande
Guest
|
| Posted: Thu Jul 31, 2008 7:40 pm Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
On 2008-07-31 11:05:17 -0300, Lurfys Maw <invalid@invalid.invalid> said:
[quote]On Thu, 31 Jul 2008 12:22:42 +0200 (CEST),
spellucci@fb04373.mathematik.tu-darmstadt.de (Peter Spellucci) wrote:
In article <jjk2949aog5ag5517njt6vuujqedr3jc4b@4ax.com>,
Lurfys Maw <invalid@invalid.invalid> writes:
Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those?
both methods are in use:
in the sense of statistics the least squares method requires the
assumption of identically normally distributed random errors in the
data (not the regressors) with zero mean.
the gauss-markov theorem then states thta the expectation values of the
computed parameters are the "true" ones.
In practice one seldom bothers about these assumptions but takes
this approach because it leads to a differentiable optimization
problem which can be solved efficiently.
The least squares method however is quite sensitive to outliers in the
data.
for "robust" regression the sum of absolute deviations is used, but this is
originally a nonsmooth problem. transformation to a smooth
problem results in a large constraint optimization problem which to solve
is much harder.
Whew. Is there any way to put that in "lay" terms?
What>s special about the power of "2" vs "1" or "3" or "1.5" or "pi"
or "e" or ...?
[/quote]
If you assumed the errors were scaled versions of uniform on -1 to +1
then you would be doing regression in the L_inifinty norm. Often called
Chebycheff (or one of many many other spellings!) fitting.
L_2 is Gaussian. Analysis is simple as derivatives can be calculated.
L_1 is Laplacian. Analysis requires keeping track of + or - of absolute
values so leads to linear programming style methods.
[quote]Why not least cubes? That would be even more sensitive to outliers,
no? Why is squares just the right amount of sensitive, whereas "3" is
too sensitive and "1" not enough?
You probably answered this, but I can>t grasp it.[/quote] |
|
| |
|
| Back to top |
Lurfys Maw
Guest
|
| Posted: Fri Aug 01, 2008 1:29 am Post subject: Re: Why is it "least squares" and not "least absolute diffre |
|
|
On Thu, 31 Jul 2008 11:33:11 -0700 (PDT), vontressms@cs.com wrote:
[quote]On Jul 31, 12:54 am, Lurfys Maw <inva...@invalid.invalid> wrote:
Why is the "least squares" method used in calculating a regression
value rather than a "least absolute differences"?
Why are the differences squared? Why not take the absolute values of
the differences and sum those?
The reason least squares is used more is historical. It has a simple
derivation for the values of the parameters that minimize the error
sum of squares based on differential calculus. This is how Gauss
discovered it many years ago. Simply put, you can solve for the
regression paramters that minimize the sum of squared residuals using
simple calculus.
Also, the objective function can also be put into the exponent of the
normal distribution. From there, you can show that the least squares
estimate of the regression parameters is also the maximum likelihood
estimate for the normal distribution. From that, you can derive
distributional results for the best regression coefficients.
Basically, least squares is associated with a lot of nice analytical
results. Least absolute differences does not have a simple analytical
solution like least squares. Also, the distribution of the least
absolute difference solution to the "regression" equations is not as
easy.
[/quote]
Thank you. This is probably what the others said, as well, but I
understand it the way you say it. |
|
| |
|
| Back to top |
|
