Reinforcement Learning

In supervised learning, we supply examples of the answer we want. For our robot, that could mean labeling each junction with the turn it should take. But sometimes it is easier to judge an outcome than to describe every decision needed to reach it. We can recognize a successful delivery without writing a rule for every possible route.

Reinforcement learning gives us a way to learn from those outcomes. The learner acts, receives feedback, and adjusts its future choices. We need a few distinct components to make that loop work:

• Agent: the decision maker. Here, it is the robot and the rule that chooses its route.
• Environment: what the agent interacts with. Our simulator responds to a route choice with an outcome.
• Observation: the information available when deciding. Here, the robot always sees the same junction. In a larger task it might see its location, nearby obstacles, and battery level.
• Action: a choice the agent can make. Our choices are Left, Straight, and Right.
• Reward: a numerical score supplied after an action. It expresses which outcomes the task designer values.
• Policy: the rule that turns an observation into a choice. Our policy assigns each action a probability, then samples one.

The environment has a state: the information needed to describe its situation and how it can change. The observation may reveal all of that state or only part of it. Our fixed junction removes this complication so we can concentrate on learning one choice.

One attempt from start to finish is an episode. Our episode contains one route choice and one reward. This is a bandit problem, a simple setting with no later decisions to plan.

The table in the live example exposes all three payoffs so you can understand the environment. The learner does not use the table to look up the best route. It samples an action and receives only that action's reward. The policy then changes through the feedback from repeated attempts.

Those payoffs are the whole of this environment. Each route leads to one outcome, and the task designer has given each outcome a score.

The scores say what we value. A quick delivery scores well above a slow one, and a blocked route scores below zero. Nothing in them tells the robot which turn to take. It has to find that out by turning.

Here is what the first five episodes of the live example look like at its default settings. Each row is one complete trip round the loop. The policy offers its probabilities, one action is sampled, the environment answers with that action’s reward, and the probability of the sampled action changes.

Read down the last two columns. A positive reward raised the probability of the route that earned it, and the reward of one raised it further than the rewards of 0.2 did. The negative reward lowered it. Episode 1 taught the robot nothing about what Left or Right would have paid, because it took neither. It learned only that Straight, tried once, went well. Part 3 calculates the size of these changes.

Trying uncertain actions is exploration. Using what has already worked is exploitation. Sampling lets this example explore while increasingly favoring rewarding routes. If a probability becomes tiny, however, that route may rarely be tried. Random sampling alone does not guarantee enough exploration.

The sixty-episode run shows this in small form. By its end Right has a probability of 0.053, after being sampled 9 times. Here that is the right outcome, because Right pays the same poor reward every time it is taken. In a task whose rewards vary from one try to the next, a route that did badly on its first few tries would be treated the same way, and at a probability that small it would seldom get another chance.

Now imagine the robot must make several moves before delivering its package. Each move uses energy, so it receives a small negative reward. Reaching the destination brings a larger positive reward. Judging only the next reward could teach it to avoid moving, even when moving makes the delivery possible.

The return combines rewards from a decision onward. A discount factor, called gamma, controls how much weight a reward keeps as it lies further into the future. Gamma of one keeps the full value; smaller values give less weight to more distant rewards.

For a separate three-step example, suppose the observed rewards are negative one, negative one, and five. Use a discount factor of 0.9. Work backward from the final reward:

Rewards: [-1, -1, 5]
Discount γ = 0.9

G₂ = 5
G₁ = -1 + (0.9 × 5) = 3.5
G₀ = -1 + (0.9 × 3.5) = 2.15

The first move had an immediate cost but a positive return. It helped begin a sequence that reached the destination. This is why a learner may reinforce an action even when its immediate reward is negative.

Whether it does so depends on the discount. The same three rewards give the first move a different return at each value of gamma, and only the weights change. Work the undiscounted case and a heavier discount the same way, backward from the final reward:

γ = 1:    G₁ = -1 + (1 × 5) = 4
          G₀ = -1 + (1 × 4) = 3

γ = 0.5:  G₁ = -1 + (0.5 × 5) = 1.5
          G₀ = -1 + (0.5 × 1.5) = -0.25

With no discount the delivery counts in full and the first move is worth three. At 0.5 the delivery two steps away is worth a quarter of its five by the time it reaches the first decision, and 1.25 does not cover the immediate cost of one plus the second cost discounted to a half. The return turns negative, and the same successful episode would now discourage the move that began it. So the discount is not a detail of the arithmetic. It decides how far ahead a cost can be repaid, and with that whether this first move is reinforced or discouraged.

A good final result does not reveal exactly which earlier choices made it happen. A useful turn and an unnecessary detour can occur in the same successful episode. Working out which decisions deserve credit is the credit-assignment problem.

Reward-to-go gives each action feedback from the rewards that follow it. It is a learning signal across many attempts, not proof that every action in one successful attempt was useful. The live bandit example avoids this difficulty deliberately: its only action receives its only reward immediately.

We now need a precise rule for changing the probabilities. Give each route a score, called a logit. Softmax converts the scores into positive probabilities that sum to one. Raising one score relative to the others raises that route's probability.

Our initial logits are all zero. The order throughout these calculations is Left, Straight, Right.

pⱼ = exp(zⱼ) / Σₖ exp(zₖ)

z = [0, 0, 0]
exp(z) = [1, 1, 1]
p = [1/3, 1/3, 1/3]

Suppose we sample Straight and receive a reward of one. Because the episode has only one step, its return is also one. We want an update that increases the probability of this sampled action.

A policy-gradient method changes the policy parameters in a direction estimated to improve expected return. The REINFORCE rule weights the gradient of a sampled action's log probability by its return. This allows learning without differentiating through the environment or its reward function.

We can subtract a baseline from the return. It provides a reference for judging the outcome. When the baseline estimates the usual return from this state, the difference tells us how much better or worse this attempt was than expected. That difference is an estimate of the advantage.

Our example uses a fixed zero baseline to keep the calculation visible. Positive rewards therefore reinforce the chosen action and negative rewards discourage it. A suitable baseline can reduce noise in the gradient estimate; it must not depend on which action was sampled for this update. See the policy-gradient derivation and baseline explanation.

A = G - b

For action a and logit j:
∂ log(pₐ) / ∂ zⱼ = 1[j = a] - pⱼ

gⱼ = A × (1[j = a] - pⱼ)
z_new = z_old + η g

G: observed return
b: baseline
A: return relative to the baseline
η: learning rate
1[j = a]: one for the selected action, zero otherwise

log(pₐ) = zₐ - log(Σₖ exp(zₖ))

∂ log(pₐ) / ∂ zⱼ = 1[j = a] - exp(zⱼ) / Σₖ exp(zₖ)
                 = 1[j = a] - pⱼ

The indicator creates a positive direction for the selected action and negative directions for the others. The advantage sets the sign and size of that feedback. The learning rate controls the size of the parameter change. This is gradient ascent, so we add the update.

Use the sampled Straight action, a zero baseline, and a learning rate of 0.1:

A = 1 - 0 = 1

Selected-action indicator = [0, 1, 0]
g = 1 × ([0, 1, 0] - [1/3, 1/3, 1/3])
  = [-1/3, 2/3, -1/3]

z_new = [0, 0, 0] + 0.1 × [-1/3, 2/3, -1/3]
      = [-1/30, 1/15, -1/30]

exp(z_new) ≈ [0.967216, 1.068939, 0.967216]
Sum of exponentials ≈ 3.003371
p_new ≈ [0.322043, 0.355913, 0.322043]

Straight is now more likely, and the other routes remain possible. The reported probabilities use full-precision logits; intermediate exponentials are rounded for display. One update reflects one experience. It does not establish which route will be best across all future situations.

Suppose a language model writes a small function in response to a prompt. We can run tests on the completed function and score the outcome. Those tests tell us something about the result without providing the exact response the model should have written.

The model is the agent. The prompt and tokens generated so far form its observation. Its next-token distribution is the policy, and a sampled token is an action. The response forms a sequence of actions. A test runner or another evaluator supplies feedback, possibly only when the response is complete.

The policy-gradient idea connects that feedback to the probabilities of the tokens the model chose. The model does not need a derivative of the test runner. It needs the score and the derivatives of its own log probabilities. Assigning credit over a whole response is much harder than updating our three route scores.

Supervised fine-tuning supplies a demonstrated response and trains the model to predict its tokens. Reinforcement learning can instead sample a response from the current policy and use an assessment of its outcome. The assessment might come from executable checks or from a model trained to predict human judgments.

In the InstructGPT approach to reinforcement learning from human feedback, or RLHF, human demonstrations first support supervised fine-tuning. People then compare generated responses. Those comparisons train a reward model, and its scores guide policy optimization. A penalty relative to a reference policy discourages excessive drift during that optimization. See Training language models to follow instructions with human feedback.

A reward model predicts a judgment; it does not establish truth. A policy can discover responses that receive high scores while missing the intended goal. Executable checks also cover only the properties they test. Evaluation must examine whether higher reward corresponds to better results on the actual task.

LoRA describes a way to represent trainable parameter changes while keeping a base matrix fixed. Reinforcement learning describes learning from actions and rewards. You can train LoRA factors using a supervised loss, or use them as the trainable parameters in a reinforcement-learning setup. Choosing an adapter does not determine the feedback signal.

The same distinction applies to algorithms. Proximal Policy Optimization, or PPO, addresses how to update a policy from sampled experience. Its clipped objective limits the incentive for some large probability-ratio changes, which can make optimization more stable. It does not guarantee that every update improves behavior. See Proximal Policy Optimization Algorithms.

Direct Preference Optimization, or DPO, is a related way to adapt a language model using preferred and rejected response pairs. It optimizes a preference objective directly, without a separately trained reward model and the online sampling loop used by the RLHF approach above. It belongs in the broader preference-learning picture, but it is not the REINFORCE loop implemented on this page. See Direct Preference Optimization.

The SDK stores three trainable logits. Each episode samples from the current policy, obtains the selected route's payoff, and applies one update. This snippet uses the same rewards and default settings as the live example.

import numpy as np
from oop_ml.numpy.modern import SoftmaxPolicy, discounted_returns

policy = SoftmaxPolicy(np.zeros(3))
run = policy.train_bandit(
    rewards=np.array([0.2, 1.0, -1.0]),
    episodes=60,
    learning_rate=0.1,
    seed=0,
)

print(run.initial)
print(run.final)
print(run.counts)
first = run.updates[0]
print(first.action, first.return_value)
print(first.before, first.gradient, first.after)

# Separate three-step return calculation, not part of the bandit.
print(discounted_returns(np.array([-1.0, -1.0, 5.0]), gamma=0.9))

Run as written, the sixty episodes sample Straight 38 times, Left 13 times and Right 9 times, and the final probabilities come out near 0.120, 0.826 and 0.053 from the equal thirds they started at. The first recorded update is the one worked in Part 3, since with this seed the first sampled action is Straight and its reward is one, so the probability of Straight moves from a third to about 0.356. Seeds one, two and three end with Straight at 0.846, 0.840 and 0.850, a different sequence of experiences arriving at the same preference. The last line prints the 2.15, 3.5 and 5 of Part 2.

With zero episodes, the run reports the starting probabilities and no observations. Repeating a fresh policy with the same seed reproduces the run. Calling training again on the same policy continues from its learned logits.

The example performs real policy updates in a deliberately small environment. It has one observation, deterministic rewards, and one action per episode. It does not train a navigation network, solve delayed credit assignment, or fine-tune a language model. The discounted-return helper illustrates the longer-episode calculation separately.

Policy gradients are one family of reinforcement-learning methods. Value-based methods, such as Q-learning, instead estimate the return associated with an action in a state and use those estimates to choose. Actor-critic methods combine a policy with a learned value estimate. Each adds machinery to address problems that our single-junction example removes.

For a larger task, inspect reward design, exploration, the cost of collecting experience, and performance on situations outside the training runs. A higher reward on this fixed junction measures progress on this fixed junction.