Understanding LSTMs – Part 3: Breaking Down the First Step

In the previous article , we just had a glimpse of long-term and short-term memory paths. Let us assume the previous long-term memory is 2. And the previous short-term memory is 1. Now we plug in some input values into the white line. Now that we have plugged in the numbers, let us do the math to see what happens in the first stage of the LSTM unit. We start with the short-term memory, which is 1, and multiply it by its weight, 2.70. Then we multiply the input value, 1, by its weight, 1.63. Next, we add these two terms together: Finally, we add the bias term 1.62: This value, 5.95, becomes the input to the sigmoid activation function, which corresponds to the x-axis value. Now we plug this into the sigmoid function: So the y-axis value is approximately 0.997. Lastly, we multiply this output by the long-term memory, which is 2: So, in this first stage of the LSTM unit, the long-term memory is reduced slightly. We will continue exploring the next stages in the coming articles Looking for