Understanding RNNs – Part 4: The Vanishing and Exploding Gradient Problem

In the previous article , we understood the concept of unrolling a network. In this article, we will examine the downsides of doing this in real-world scenarios. The problem when unrolling One big problem is that the more we unroll a recurrent neural network, the harder it is to train. This problem is called the vanishing or exploding gradient problem. In this example, the vanishing or exploding gradient problem is related to the weight along the connection that we copy each time we unroll the network. To make this easier to understand, we will ignore the other weights and biases and focus only on w2. As we did earlier, when we optimize a neural network using backpropagation, we first find the derivatives, or the gradients, for each parameter. Then we plug those gradients into the gradient descent algorithm to find the parameter values that minimize the loss function, such as the sum of squared residuals. Understanding Exploding Gradients Now we will see how a gradient can explode. In