Support Vector Machines — Lecture series — The SVM Lagrangian problem

In the previous post, we talked about how to apply the concept of Lagrange multipliers to solve optimisation problems. In this post, we will be talking about how to apply the concept of Lagrange multipliers to solve an SVM optimisation problem.

Learning objective:

Understand how to apply Lagrange multipliers to solve SVM optimisation problems.

Main question:

How would you apply the concept of Lagrange multipliers to solve the SVM optimisation depicted in Fig. 1 below:

Fig. 1

Well, the first step would be to introduce a Lagrange function. Like what is depicted in Fig. 2 below:

Fig. 2

You will notice that in the Lagrange function that we have introduced in Fig. 2, there is one Lagrange multiplier alpha for each constraint function.

The next step here would be to solve for when the derivative of the Lagrange function is equal to zero since this is how we find the minimum of the function. This step is however not so easy to solve. In the next post, we will look at why this step is not easy to solve and how we can apply the duality principle to overcome it.