Understanding LSTMs – Part 4: How LSTM Decides What to Forget

In the previous article , we completed the first part of the LSTM and obtained the result from the calculation. Let us continue. Earlier, when the input was 1, we obtained the following result: Now, if we change the input to a relatively large negative number, such as −10, then after calculating the x-axis value, the output of the sigmoid activation function will be close to 0. The long-term memory will be completely forgotten, because anything multiplied by 0 is 0. Since the sigmoid activation function converts any input into a value between 0 and 1, its output determines what percentage of the long-term memory is retained. So, the first stage of the LSTM determines what percentage of the long-term memory is remembered. This part is called the forget gate. Now that we understand what the first stage does, let us explore the second stage. In the second stage, the block on the right combines the short-term memory and the input to create a potential long-term memory. The block on the lef