Recurrent Layers

A measurement in a sequence may only make sense in light of what came before it. Recurrent layers carry a state between steps, and gates give us more control over what information is kept.

10 sections · 10 questions · 3 coding challenges

1 / 10

The arrow keys move between sections too.

Could the Previous Step Leave a Useful Memory?

BOTIE

A sound or word often means something different depending on what preceded it. If we process each observation independently, we throw away that context before making the next prediction. We need a representation that can carry something forward.

Elman's 1990 Finding Structure in Time explored recurrent connections as a way to give a network a changing internal memory. Instead of representing time only by placing a fixed window of observations side by side, the network's state could reflect the sequence it had processed.

At each step, a recurrent layer combines the new input with its previous hidden state and produces an updated state. The same parameters are reused along the sequence. This lets history influence the present computation without storing the entire past explicitly, but information can be weakened or overwritten. Longer-range memory requires careful architecture and training rather than following automatically from recurrence.

Sources for this history

Elman (1990), Finding Structure in Time

How Do We Remember What Came Before?

The latest temperature reading does not say whether the temperature has been rising or falling. Earlier readings can matter even when the current reading is identical.

A recurrent layer carries a state from one step to the next so earlier information can influence the current response.

Compare histories with the same ending

Both sequences finish at the same temperature, but one is rising and the other falling. A model seeing only the last reading cannot distinguish them.

Rising

Falling

Illustrative temperature histories.

1 of 4

The state is the link between observations

We can now calculate one state update, then compare the recurrent variants and their gates.

The full experiment distinguishes outputs at each position from the final state, and explains how gradients travel across time.

Explore the full example

Start with the simple recurrent cell. Follow the first input through the later steps, which all receive zero. Then select LSTM and change the forget gate bias. Compare the cell memory with the visible hidden state.

A signal arrives at the first step. Later inputs are zero. These cells use chosen weights to expose their operations; this is not a trained performance comparison.

Cell

Steps: 6First input: 1Forget gate bias: 1

Calculating the example…

Part 1. Carry a State from One Step to the Next

1. Reuse one update rule

A dense layer answers each row from that row alone. Hand it the readings of a sequence one at a time and its fourth answer knows nothing about the first three. For a sentence, a melody or a week of temperatures that is the wrong arrangement, because what a reading means depends on what came before it.

What is missing is something carried from one step to the next. A recurrent layer keeps a small vector called the hidden state. At every step it reads the current input together with its own previous state, calculates a new state, and hands that state to the next step.

At the first step, the SDK starts the hidden state at zero. We concatenate the current input with that state, calculate a weighted sum and bias, then apply tanh. The result becomes the next hidden state. The same weights are reused at every step.

combined_t = concatenate(input_t, hidden_(t−1))
hidden_t = tanh(W combined_t + b)

W holds one weight for every input value and one for every state value, and b is a bias. The weights on the input decide how strongly a new reading enters the state. The weights on the previous state, called the recurrent weights, decide how much of what was already there carries on. The tanh function then keeps every state value between minus one and one, so the state cannot grow without limit however long the sequence runs.

The state has a fixed width. It is a learned summary, not a copy of all earlier readings. A model can use the final state to predict something about the sequence, or produce an output at every step.

One signal, followed by zeros

The simple cell in the playground has one state value. Its input weight is one, its recurrent weight is one half, and its bias is zero.

Initial state = 0
First input = 1
First state = tanh((1 × 1) + (0.5 × 0))
            = tanh(1) ≈ 0.7616

Second input = 0
Second state = tanh((1 × 0) + (0.5 × 0.7616))
             ≈ 0.3634

The first reading still influences the second state even though the new input is zero. With these chosen weights, its influence becomes smaller as we continue. Different weights can produce different behavior.

Carry the same rule through all six steps of the playground’s default sequence, with every later input at zero.

Step	Input	Hidden state
1	1	0.7616
2	0	0.3634
3	0	0.1797
4	0	0.0896
5	0	0.0448
6	0	0.0224

The simple cell over six steps, as the playground’s first table shows it.

Each state is a little under half the one before it. The half is the recurrent weight, and the little under is tanh, which returns slightly less than it is given. After six steps the state is 0.0224, so the cell still records that something arrived, though only faintly.

2. Understand why learning through many steps can be difficult

To learn from an early input, the final prediction error must influence the weights used at that early step. Backpropagation passes through every intervening state update. Multiplying many small derivatives can make the resulting gradient very small; other weights can make it grow excessively.

The quantity to watch is a derivative, how far the final state would move if one input moved a little. Going back one step multiplies that derivative by two numbers. One is the recurrent weight, since that is how much of the earlier state entered the later one. The other is the slope of tanh at the state in between, which is one minus the square of that state and is never more than one.

How far each input can still move the final state

The playground reports this derivative for every input of the simple cell, in the panel that opens the gates and the gradient. The last input reaches the final state through one tanh. Each earlier input has one more step to cross.

derivative for input 6 = 1 − 0.0224² ≈ 0.9995
derivative for input 5 ≈ 0.9995 × 0.5 × (1 − 0.0448²) ≈ 0.4987
derivative for input 4 ≈ 0.4987 × 0.5 × (1 − 0.0896²) ≈ 0.2474
...
derivative for input 1 ≈ 0.0519 × 0.5 × (1 − 0.7616²) ≈ 0.0109

Input step	Derivative of the final state
1	0.0109
2	0.0519
3	0.1197
4	0.2474
5	0.4987
6	0.9995

The simple cell over six steps. Each row is how far the sixth state moves for a small change to that input.

The last input moves the final state almost one for one, and the first moves it about ninety times less. The step from input 2 back to input 1 costs more than the others, because the first state is 0.7616, where tanh is flatter and its slope is about 0.42. Set the playground to twelve steps and the derivative for the first input is about 0.0002.

Nothing fails when this happens. The cell runs and an error can still be measured on its final state. But the early inputs have almost no say in that state, so the error can teach the weights very little about them.

This gives us a reason to change the update rule. We want a controllable route for keeping information, instead of repeatedly transforming all of it in the same way.

Part 2. Controlling What the State Retains

3. Give an LSTM a separate memory

An LSTM, or long short-term memory cell, carries a cell memory as well as a hidden state. A gate is a learned number between zero and one that controls how much of a quantity passes through. There are three gates and one candidate to understand:

The forget gate controls how much previous cell memory remains.
The input gate controls how much new candidate information is added.
The candidate supplies proposed new memory values.
The output gate controls how much of the transformed cell memory becomes the hidden state.

Each is calculated from the current input and previous hidden state. Sigmoid produces the gate values; tanh produces the candidate. The SDK calls the input gate "keep" and the output gate "show."

c_t = f_t ⊙ c_(t−1) + i_t ⊙ candidate_t
h_t = o_t ⊙ tanh(c_t)

The multiplication symbol denotes matching-coordinate products. The memory update adds retained information and new information. Along its direct memory route, the derivative is controlled by the forget gate.

Choose what to retain and what to add

Suppose the previous memory is two, the forget gate is eight tenths, the input gate is one quarter, and the candidate is four tenths.

Retained memory = 0.8 × 2 = 1.6
New contribution = 0.25 × 0.4 = 0.1
Updated memory = 1.6 + 0.1 = 1.7

We have kept some existing information while adding a smaller new contribution. These are illustrative values. In a trained LSTM, the gates are calculated separately for each step and coordinate.

The playground’s LSTM, two steps by hand

The playground’s cell has one memory value and deliberately simple weights. The candidate reads the input with a weight of one and ignores the previous hidden state. Every gate has zero weights, so each gate is the sigmoid of its bias and takes the same value at every step. The input gate and the output gate have a bias of zero, which gives one half. The forget gate has the bias the slider sets, which is one by default.

forget gate = sigmoid(1) ≈ 0.7311
input gate = output gate = sigmoid(0) = 0.5

Step 1, input 1, previous memory 0:
candidate = tanh(1) ≈ 0.7616
memory = (0.7311 × 0) + (0.5 × 0.7616) ≈ 0.3808
hidden state = 0.5 × tanh(0.3808) ≈ 0.1817

Step 2, input 0, previous memory 0.3808:
candidate = tanh(0) = 0
memory = (0.7311 × 0.3808) + (0.5 × 0) ≈ 0.2784
hidden state = 0.5 × tanh(0.2784) ≈ 0.1357

After the first step nothing new arrives, so each step multiplies the memory by the forget gate and does nothing else to it. The memory runs 0.3808, 0.2784, 0.2035, 0.1488, 0.1088 and 0.0795 across the six steps. The hidden state is a second, smaller number read off the memory through the output gate. Unlike the simple cell’s state, the memory does not pass through tanh on its way to the next step.

Forget gate bias	Forget gate	Memory after step 2	Memory after step 6
−3	0.0474	0.0181	0.0000
0	0.5000	0.1904	0.0119
1	0.7311	0.2784	0.0795
3	0.9526	0.3627	0.2987

The same first input of one at four settings of the playground’s forget gate bias.

The memory after the first step is 0.3808 at every setting, because the forget gate multiplies the previous memory and at the first step that is zero. What the gate decides is how much of it is still there later. At a bias of three, 0.2987 of the 0.3808 remains after six steps. At minus three, 0.0181 remains after two.

A trained cell calculates each gate afresh from the input and the previous hidden state, so it can hold a gate near one for one stretch of a sequence and let it drop for another. The playground fixes the gates so that one slider moves one gate.

4. A GRU uses one carried state

A gated recurrent unit, or GRU, combines memory and hidden state into one vector. Its reset gate controls how much previous state is used to construct the candidate. Its update gate controls the mixture of previous state and candidate.

r_t = sigmoid(W_r [x_t, h_(t−1)] + b_r)
z_t = sigmoid(W_z [x_t, h_(t−1)] + b_z)
candidate_t = tanh(W_c [x_t, r_t ⊙ h_(t−1)] + b_c)
h_t = (1 − z_t) ⊙ candidate_t + z_t ⊙ h_(t−1)

This is the SDK's convention: a larger update gate retains more of the previous state. Some implementations use the complementary convention. The reset operation here happens before the candidate's recurrent multiplication.

The playground’s GRU, two steps by hand

The playground’s GRU has one state value. Its reset gate and its update gate have zero weights and zero biases, so both are one half at every step. Its candidate reads the input with a weight of one and the reset previous state with a weight of one half.

reset gate = update gate = sigmoid(0) = 0.5

Step 1, input 1, previous state 0:
candidate = tanh((1 × 1) + (0.5 × 0.5 × 0)) = tanh(1) ≈ 0.7616
state = (0.5 × 0.7616) + (0.5 × 0) ≈ 0.3808

Step 2, input 0, previous state 0.3808:
candidate = tanh((1 × 0) + (0.5 × 0.5 × 0.3808)) ≈ tanh(0.0952) ≈ 0.0949
state = (0.5 × 0.0949) + (0.5 × 0.3808) ≈ 0.2379

The last line of each step is the mixture. Half of the new state is the candidate and half is the previous state carried over unchanged, because the update gate is one half. That carried half does the job the forget gate did in the LSTM, without a separate memory to hold it. Across the six steps the state runs 0.3808, 0.2379, 0.1486, 0.0929, 0.0580 and 0.0363.

Part 3. Learning Across Time

Parts 1 and 2 ran the cells forward. Training runs them backward. An error is measured on the output, and each weight is moved according to the derivative of that error with respect to it. A recurrent layer uses the same weights at every step, so the backward pass has to visit every step, from the last to the first, and collect what each one contributes.

Two things are handed back from each step to the one before it. One is the derivative with respect to the hidden state. In an LSTM the other is the derivative with respect to the cell memory, which travels by its own route. How much of a derivative survives the journey decides how much an early input can teach the weights, and the playground measures it.

The route back through the LSTM memory, measured

Select the LSTM in the playground with the forget gate bias at one, and open the gates and the gradient. The second table there is the derivative of the final hidden state with respect to each input. Start at the last input and work backward.

derivative for input 6 = 0.5 × (1 − tanh²(0.0795)) × 0.5 ≈ 0.2484
derivative for input 5 ≈ 0.2484 × 0.7311 ≈ 0.1816
derivative for input 4 ≈ 0.1816 × 0.7311 ≈ 0.1328
derivative for input 3 ≈ 0.1328 × 0.7311 ≈ 0.0971
derivative for input 2 ≈ 0.0971 × 0.7311 ≈ 0.0710
derivative for input 1 ≈ 0.0710 × 0.7311 × (1 − 0.7616²) ≈ 0.0218

The first line is the output gate, the slope of tanh at the final memory, and the input gate. After that, each step further back multiplies by the forget gate and by nothing else, because in this cell the only way an early input reaches the final state is through the memory. The first input pays one more factor, the slope of tanh at its own candidate of 0.7616. The later candidates sit at zero, where that slope is one.

Now move the forget gate bias. At three the gate is 0.9526 and the derivative for the first input rises from 0.0218 to 0.0754. At zero the gate is one half and it falls to 0.0033. In the simple cell of Part 1 the factor paid at each step was the recurrent weight times the slope of tanh, and the cell had no way to set that factor apart from the state it was calculating. Here the factor is a gate, a number the cell can learn to hold near one where something is worth keeping.

›Backpropagation through the repeated rule

Start at the final step and work backward. Each step receives a derivative from its output and from the state it supplied to the next step. It calculates input derivatives, parameter derivatives, and the derivatives to pass to the preceding state.

Total weight gradient = sum of each step's weight contribution
Total bias gradient = sum of each step's bias contribution

We add the contributions because every step uses the same parameters. LSTM also passes a derivative through the cell memory. The completed NumPy methods account for both the memory route and the route through the hidden state.

Keep in mind

Gates provide ways to retain information; they do not guarantee perfect memory or prevent all gradient problems. The playground uses different chosen rules to expose the mechanisms, so it does not establish that one architecture will outperform another on a trained task.

The original LSTM paper addresses learning over long time intervals. The implementation here uses a later, commonly used forget-gate form. Attention takes a different route by allowing positions to gather information directly from other positions.