How to derive the least squares formula

　・In Japanese
＜Premise knowledge＞
　・Variance, Covariance
　・the least squares formula

■simple regression equation by least squares method
As explained here, the formula of the simple regression equation obtained by the least squares method is as follows. And I will explain how to derive it.



Since the least squares method finds A and B that minimize the sum of squares of the residuals from the regression equation, it is as follows.



Partial differential of the above equation with A and B.



(2) is modified as follows. Each term is divided by 2 for simplicity.



Now you can find B. Dividing the sum of y_i and x_i by n means the average.
Next, (1) is transformed as follows. This also divides each item by 2.



Substitute (3) into the above equation.



From the variance formula.



Therefore,



Now A can also be derived.

■multiple regression equation by least squares method
The formula for two variables (w, x) is as follows. Since the concept of the derivation method is the same as for simple regression, the detailed calculation process is omitted.



Eq. (4) can be easily understood by expressing it as a matrix as shown below, and this matrix is called a variance-covariance matrix.



■three-variable multiple regression equation by least squares method
It will be as follows. Even if the number of variables increases, it can be handled in the same way.

List of related articles TOP MENU Probability / Statistics Statistics/analysis Expected Value Variance, Covariance standard deviation Root Mean Square least squares formula Weighted/Moving average Gaussian process regression Kernel function Basis functions Probability Joint probability Conditional probability Weibull distribution