Support Vector Machines — Lecture series — Lagrange multipliers part 1
In the previous post, we further developed the optimisation problem statement for finding the geometric margin for a given set of linearly separable points. In this post, we will be looking at a method that enables us to solve an optimisation problem statement that is characterised by an equality constraint.
Learning objective:
Understand the concept of Lagrange multipliers.
Main question:
Given a “maximize” or “minimize” optimisation problem statement that is subject to an equality constraint such as the one in Fig. 1 below:
How do we solve for the unknown variable (x)?
Well, we can apply the concept of Lagrange multipliers to help us out.
The concept of Lagrange multipliers was invented by a french mathematician called Giuseppe Lodovico Lagrangia also known as Joseph Louis Lagrange. It hinges on the idea that the point of optimisation of the function being optimised is located where the gradient at that point points in the same direction as the gradient of the constraint function.
Okay, let’s take a step back and break this down again in the context of the example optimisation problem statement in Fig. 1
In applying the Lagrange multipliers to Fig. 1, we can state that the minimum of f is found when its gradient points in the same direction as the gradient of g. In other words when:
In the subsequent posts, we will be plunging further into this concept.