rambo

LR（二、实践）

2015年8月20日 Machine Learning 0 Comments

任务1：根据每个学生两场考试的成绩来预测他（她）是否能够考入大学？（可以看出数据是线性可分的）

训练样本数据矩阵X\in R^{100\times 2}(100个训练样本，2维特征）可视化，黑点表示正样本，黄点表示负样本。

QQ截图20150820163717

 

LR中的loss function

\begin{split}l(\beta)&=\frac{1}{N}\sum_{i=1}^{N}\{y_{i}log p(x_{i};\beta)+(1-y_{i})log(1- p(x_{i};\beta)\}\\&=\frac{1}{N}\sum_{i=1}^{N}\{y_{i}\beta^{T}x_{i}-log(1+exp^{\beta^{T}x_{i}})\}\end{split}

损失函数的梯度

极大化对数似然，对\beta的每一个分量\beta_{j}求偏导，j=1,...,p+1，显然其维数与\beta维数一致。

QQ截图20150820021628

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
%% ============ Part 2: Compute Cost and Gradient ============
 
%  In this part of the exercise, you will implement the cost and gradient
 
%  for logistic regression. You neeed to complete the code in
 
%  costFunction.m
 
 
%  Setup the data matrix appropriately, and add ones for the intercept term
 
[m, n] = size(X);//m为样本维数，
 
% Add intercept term to x and X_test
 
X = [ones(m, 1) X];//这里增加一维特征 （数值为1）作为截距，只是为了编程方便
 
% Initialize fitting parameters
 
initial_theta = zeros(n + 1, 1);  //权重系数向量，同理也增加为3维
 
% Compute and display initial cost and gradient
 
[cost, grad] = costFunction(initial_theta, X, y);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
function [J, grad] = costFunction(theta, X, y)
%   J = COSTFUNCTION(theta, X, y) computes the cost of using theta as the
 
%   parameter for logistic regression and the gradient of the cost
 
%   w.r.t. to the parameters.
 
 
% Initialize some useful values
 
m = length(y); % number of training examples
 
% You need to return the following variables correctly
 
J = 0;
grad = zeros(size(theta));
 
% ====================== YOUR CODE HERE ======================
 
% Instructions: Compute the cost of a particular choice of theta.
 
%               You should set J to the cost.
 
%               Compute the partial derivatives and set grad to the partial
 
%               derivatives of the cost w.r.t. each parameter in theta
 
% Note: grad should have the same dimensions as theta
 
%
 
J = sum ( -1 * y .* log ( sigmoid(X*theta)) - (1-y).*log(1-sigmoid(X*theta)));
J = J / m;
 
dif = sigmoid(X*theta) - y;
grad = (X' * dif) / m;
% =============================================================
 
end

这里我们对于权重系数向量的更新采用了Matlab自带的无约束最优化fminunc函数（可用于任意函数求最小值，将最大、最小统一为求最小值问题）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
%fminunc函数
 
%x=fminunc(fun,x0)
 
%x=fminunc(fun,x0,options)
 
%x=fminunc(fun,x0,options,p1,p2,...)
 
%[x,fval]=fminunc(fun,x0,options,p1,p2,...)
 
%[x,fval,exitflag]=fminunc(fun,x0,options,p1,p2,...)
 
%[x,fval,exitflag,output]=fminunc(fun,x0,options,p1,p2,...)
 
%[x,fval,exitflag,output,grad]=fminunc(fun,x0,options,p1,p2,...)
 
%[x,fval,exitflag,output,grad,hessian]=fminunc(fun,x0,options,p1,p2,...)
 
%说明：
 
%fun:使目标函数：
 
%options:设置优化选项参数：
 
%fval:返回目标函数在最优解x点的函数值：
 
%exitflag:返回算法的终止标志：
 
%output:返回优化算法信息的一个数据结构：
 
%grad:返回目标函数在最优解x点的梯度：
 
%hessian:返回目标函数在最优解x点的Hessian矩阵值。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
%% ============= Part 3: Optimizing using fminunc  =============
 
%  In this exercise, you will use a built-in function (fminunc) to find the
 
%  optimal parameters theta.
 
 
%  Set options for fminunc
 
options = optimset('GradObj', 'on', 'MaxIter', 400);
 
%  Run fminunc to obtain the optimal theta
 
%  This function will return theta and the cost
 
[theta, cost] = ...
fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);
 
% Print theta to screen
 
fprintf('Cost at theta found by fminunc: %f\n', cost);
fprintf('theta: \n');
fprintf(' %f \n', theta);

1
2
3
4
5
Cost at theta found by fminunc: 0.203506
theta:
-24.932998
0.204408
0.199618

根据优化得到的权重参数画出分界面或分界线。
QQ截图20150820170342

分类函数：

1
2
3
4
5
6
7
8
9
% J = sigmoid(z) computes the sigmoid of z.
 
prob = sigmoid([1 45 85] * theta);
 
function g = sigmoid(z)
g = zeros(size(z));
g = 1 ./ ( 1 + exp(-1.*z) );
 
end

p = predict(theta, X);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
function p = predict(theta, X)
%PREDICT Predict whether the label is 0 or 1 using learned logistic
 
%regression parameters theta
 
%   p = PREDICT(theta, X) computes the predictions for X using a
 
%   threshold at 0.5 (i.e., if sigmoid(theta'*x) >= 0.5, predict 1)
 
 
m = size(X, 1); % Number of training examples
 
p = zeros(m, 1);
h = sigmoid(X*theta);
p( find(h>=0.5) ) = 1;
 
end

准确率为89%。

 

http://bbs.vsharing.com/Technology/CloudComputing/1743704-1.html

http://www.cnblogs.com/jerrylead/archive/2011/03/05/1971903.html

http://blog.csdn.net/viewcode/article/details/8794401

http://blog.csdn.net/sprintfwater/article/details/8948349#comments

http://blog.csdn.net/yuwei19840916/article/details/3245107