Natural Language Processing — Neural Networks and Neural Language Models Lecture series — The XOR problem
In this post, we will be learning about the XOR problem and how it can be solved.
Consider the following image in Fig. 1:
The XOR problem is that, we can formulate perceptrons that could take in as inputs the values of x1 and x2 for the operations of AND and OR and generate their corresponding y outputs but it would be impossible to formulate a perceptron that could take in as inputs the values of x1 and x2 and then generate their corresponding y outputs for the operation of XOR.
Note: A perceptron is a neural unit that has a binary output and does not have a non-linear activation function. In simple terms, a perceptron is a linear classifier. The output of a perceptron is ‘0’ or ‘1’ and it is computed as follows:
Where w is a vector of the weights, x is a vector of the input values and b the bias term. The value of the bias term b is usually placed in the position of the first index of the vector x and its corresponding weight is usually also placed in the position of the first index of the vector w.
For the logical operation of AND, we can build a perceptron where the vector values of the weights w can be [-1, 1, 1] and the vector of their corresponding inputs can be [1, x1, x2].
However for the logical operation of OR, we can build a perceptron where the vector values of the weights w can be [0,1,1] and the vector of their corresponding inputs can be [1,x1,x2].
Since a perceptron is a linear classifier, for the logical operation of AND and OR, we can generate images that depict that the values classified as either a ‘0’ or a ‘1’ for these logical operations can be separated by a straight line. This is demonstrated in the image in Fig. 2 below:
However when we graphically depict the output values of the logical operator XOR, we see that we cannot generate any line to classify its output points, hence supporting the statement that a single perceptron cannot be built for this operation.
To address the XOR problem, we would need to employ the idea of neural networks. In the next post, we shall talk about how Neural Networks can be used to address the XOR problem.