Understanding Seq2Seq Neural Networks – Part 4: The Encoder and the Context Vector

In the previous article , we stopped with the problem where we wanted to add more weights and biases to fit our data. So in this article, We will add one more LSTM layer to the encoder. What this means is that the output values (the short-term memories , or the hidden states ) from the unrolled LSTM units in the first layer are used as the inputs to the unrolled LSTM units in the second layer . Just like how both embedding values are used as inputs to both LSTM cells in the first layer , both outputs ( the short-term memories ) from each cell in the first layer are used as inputs to both LSTM cells in the second layer . Initializing the Memories The only thing left to do is initialize the long-term and short-term memories . So with this, we are done creating the encoder part of the Encoder–Decoder model . The Encoder Output In this example, we have two layers of LSTMs , with two LSTM cells per layer . The encoder encodes the input sentence “Let’s go” into a collection of long-term and