Support Vector Machines — Lecture series — The limitations of the Perceptron Learning Algorithm
In the previous post, we spoke about how the update rule works. In this post, we will be talking about the limitations of the perceptron learning algorithm.
Learning objective:
Understand the limitations of the perceptron learning algorithm.
Main question:
Since it is the job of the perceptron learning algorithm to figure out a hyperplane that can correctly separate a given linearly separable dataset, when we run the algorithm on the same dataset 4 times, we will get 4 different hyperplanes that correctly separate the dataset. This is because the weights of the hyperplanes are randomly generated and the misclassified examples are randomly chosen. Such a situation is demonstrated in Fig. 1 below:
However the main question here is, which of the hyperplanes is the best for the dataset?
Well, the best hyperplane is the one that can correctly classify new data points that were not used in its formulation. This means that when a hyperplane is able to correctly classify new data points after it has been formed, we can confidently say that, that hyperplane is the most optimum hyperplane.
That being said let us reconsider the hyperplanes in Fig. 1 and then let us introduce new data points to each of those hyperplanes and introduce the results in Fig. 2:
You will see that the hyperplanes on the right-hand-side in Fig. 2 incorrectly classify new data points whiles the hyperplanes on the left-hand-side correctly classify new data points. Hence the best hyperplanes for the dataset are the hyperplanes on the right-hand-side of Fig. 2.
But the issue here is that, we will not be able to tell which hyperplane is best for any given dataset until we test it with new data. This is the limitation of the perceptron learning algorithm.
In the next post, we will talk about the Support Vector Machine’s search for the optimal hyperplane.