Support Vector Machines — Lecture series — Multi-class SVMs (part 1)

In the last series of posts we have been talking about the main ideas regarding the topic of support vector machines. In this post, we will be covering the last topic in this series by looking at the concept of multi-class SVMs and we will also be learning about the main intuitions behind some of the most popular multi-class SVM techniques.

Learning objective:

To learn about the main idea behind multi-class SVMs and also learn about the one of the different techniques that are applied in this context.

Main question:

So far we have been considering the situation in which we are to compute the best classifier for separating the points belonging to 2 classes. But the question is, how do we figure out the best classifiers when we are faced with more than 2 classes?

Multi-class SVM techniques are approaches that can be adopted to figure out the best classifiers that can be employed to separate the data points belonging to multiple classes into their respective class regions within the plane in which they lie. This is pictorially demonstrated in Fig 1. below where you have x classifiers separating the points in the plane into their respective class regions:

Fig. 1: Multi-class SVM (Image source: Chirag Goyal)

One of the techniques that can be used to generate the different classifiers is know as the one-against-all approach. In this approach, one classifier is developed per class. That is, assuming that there are 3 different classes represented in the plane, this approach would develop 3 different classifiers to classify the points belonging to each different class. Even though this approach is easy to conceptualise, it has a has some problems. One of the problems of this approach stems from the fact that it may fail to assign all of the points that exist in the plane to a specific class. This problem is demonstrated in Fig 2 below where any point that should fall within the blue shaded region of the plane would not be classified by this approach:

Fig 2: One-against-all SVM

Another problem of the one-against-all approach is that scales linearly with the number of classes that are available within the plane and as a result it can be extremely computationally expensive to execute.

In the next post, we will be looking at another multi-class SVM technique that mitigates the issues presented by this technique and arrives at a much better solution.

See you in the next post!