Exercise1: Gradient Descent Algorithm of Linear Regression Model

Model

This algorithm applies to both univariable linear regression model:
     h(x) = z0 + z1*x1
and multivariable linear regression model:
    h(x) = z0 + z1*x1 + z2*x2 + ... + zn*xn

Aka:
     h(x) = z' * x
     z = (z0, z1, ..., zn)'
     x = (x0, x1, ..., xn)', x0 = 1

Algorithm

The Gradient Descent algorithm is a vectorized implementation in Octave as below:

function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)
%GRADIENTDESCENT Performs gradient descent to learn theta
%   theta = GRADIENTDESENT(X, y, theta, alpha, num_iters) updates theta by 
%   taking num_iters gradient steps with learning rate alpha

% Initialize some useful values
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);

for iter = 1:num_iters
    % Instructions: Perform a single gradient step on the parameter vector
    %               theta. 
    %  
    %theta = theta - alpha/m * Sum{[h(x)-y] * x}

    h = X*theta;                                 %predictions h(x)
    gradients = 1/m * X' * (h-y) ;      %gradient vector
    theta = theta - alpha * gradients;

    % Save the cost J in every iteration    
    J_history(iter) = computeCost(X, y, theta);
end

end

Demo