Computer Science – 18.1 Artificial Intelligence (AI) | e-Consult

Backpropagation and Gradient Descent for Linear Regression

The goal of training a linear regression model is to minimize the Mean Squared Error (MSE) between the predicted house prices and the actual house prices. MSE is defined as the average of the squared differences between the predicted and actual values. The model's prediction is given by: yhat = w * x + b, where yhat is the predicted price, x is the size of the house, w is the weight, and b is the bias.

Gradient Descent is an iterative optimization algorithm used to find the values of w and b that minimize the MSE. It works by repeatedly adjusting w and b in the direction of the negative gradient of the MSE loss function. The gradient indicates the direction of the steepest increase in the loss function, so moving in the opposite direction leads to a decrease. The learning rate (α) controls the size of the steps taken during each iteration.

Backpropagation is the algorithm used to calculate the gradients of the MSE loss function with respect to w and b. This is crucial because the gradient descent algorithm requires these gradients to update the model's parameters.

The Chain Rule: The backpropagation algorithm relies heavily on the chain rule of calculus. Since the prediction yhat is a linear function of w and b, and the MSE is a function of yhat, we can use the chain rule to calculate the gradients.

The gradient of the MSE with respect to w is calculated as follows:

1. The derivative of the MSE with respect to yhat is 2*(yhat - y), where y is the actual house price.
2. The derivative of y_hat with respect to w is x.
3. Therefore, the gradient of the MSE with respect to w is x * (2*(y_hat - y)) = x * (2*(w*x + b - y)) = 2*x*x*w + 2x*b - 2xy

Similarly, the gradient of the MSE with respect to b is calculated as:

1. The derivative of the MSE with respect to yhat is 2*(yhat - y).
2. The derivative of y_hat with respect to b is 1.
3. Therefore, the gradient of the MSE with respect to b is 2*(y_hat - y) * 1 = 2*(w*x + b - y) = 2*w*x - 2y + 2b

The values of w and b are then updated using the following formulas:

• w = w - α * (∂MSE/∂w)
• b = b - α * (∂MSE/∂b)

This process is repeated iteratively until the loss function converges to a minimum.