Support Vector Machines — Lecture series — The update rule part 2
In the previous post, we spoke about the main idea behind the update rule. In this post, we will be looking at how to mathematically perform the update rule. Please read the part 1 of this post (https://davidsasu.medium.com/support-vector-machines-lecture-series-the-update-rule-part-1-6539ff8bb0d6)if you haven’t already done so, so that you will possess the relevant pre-requisite knowledge to understand the contents of this post.
Learning objective:
Understand how to mathematically implement the idea of the update rule.
Main question:
Given 2 vectors, w and x from the hyperplane equation, how would you correctly classify a misclassified vector of x?
In the previous post, we learnt that the correct classification of the vector x depended on the cosine of the angle between the vector x and the vector w. We also learnt that if the vector x was misclassified as a +1 when it should have been classified as a -1, we can correctly classify it by increasing the angle between the vectors of w and x. Whiles if the vector x was misclassified as a -1 when it should have been classified as a +1, we can correctly classify it by decreasing the angle between the vectors of w and x.
To increase the value of the angle between the vectors of w and x, we will have to replace the vector value of w with the difference between the vector w and the vector x (new w = old w-x). This is demonstrated in Fig. 1 below:
In Fig. 1, we see that the angle between x and w-x is larger than the angle between x and w.
However, to decrease the value of the angle between the vectors of w and x, we will have to replace the vector value of w with the summation of the vector w and the vector x (new w = old w + x). This is demonstrated in Fig. 2 below:
In Fig. 2, we see that the angle between w+x and x is smaller than the angle between w and x.
So in summary:
If the expected label is -1 but was incorrectly predicted as +1, we increase the angle by setting w = w — x
If the expected label is +1 but was incorrectly predicted as -1, we decrease the angle by setting w = w + x
In the next post, we will be looking at the limitations of the Perceptron Learning Algorithm.