Radial Basis Networks
Some patterns are useful only near particular examples. A radial basis network measures how close an input is to several reference points, then combines those local responses into a prediction.
9 sections · 5 questions · 3 coding challenges
The arrow keys move between sections too.
Could Several Local Responses Describe a Complicated Relationship?
BOTIE
Broomhead and Lowe's 1988 paper connected adaptive networks with multivariable interpolation using radial basis functions. This supplied another route to learning a function: combine responses centred on particular locations rather than relying only on successive layers of weighted sums and thresholds.
A radial response depends on distance from a centre. With a Gaussian response, nearby inputs activate it strongly and distant inputs activate it weakly. A final weighted combination joins these local responses into a prediction. Centre placement and response width determine how much the regions overlap, so they control whether the model is too local, too smooth, or a useful compromise.
Sources for this history
What If a Pattern Matters Only Nearby?
A machine may work well near a particular temperature and poorly when much colder or hotter. We want a response that is strongest near a useful region rather than increasing forever in one direction.
A radial basis unit describes a region using a centre and a width.
Pick the region a unit should respond to
The centre marks the input where a unit responds most strongly. Inputs farther away receive smaller responses under a Gaussian basis function.
Vertical: Unit response
Horizontal: Input location
Centres and widths define the local representation
The next calculation measures distance and converts it into a response, then shows how the relevant parameters can be trained.
Widths, centres, and output weights must be evaluated together. A local response is not automatically a calibrated probability.
Explore the full example
Move the input through the three centres. Then increase the width and compare how far each unit responds. The table reports the exact sample used by the plotted curves.
Calculating the example…
Part 1. Describe a Region by Its Centre and Width
1. Measure distance from a reference point
A neuron scores an input by a weighted sum, and a weighted sum has an opinion everywhere. Push the input far enough along the direction of the weights and the score keeps rising, however unlike the training data that input has become. Some patterns do not behave that way. A reading matters when it is close to a particular value and matters less the further it moves from that value, on either side.
A radial basis unit is built for that case. Where a neuron holds a direction, this unit holds a place, called its centre, and it answers with how close the input is to that place. Three things go into the answer. They are the centre, a distance from it, and a width that says how far still counts as close.
For one input coordinate, the centre is one number. For several coordinates, it is a vector with one entry per feature. We subtract the centre from the input, square each difference, and add the squared differences. This gives squared Euclidean distance.
Squaring makes equal displacements on opposite sides contribute equally. The Gaussian unit depends on distance from the centre, not on the direction from which the input approached it. That dependence on radius is why it is called radial.
2. Let width control the useful neighborhood
At the centre, the Gaussian response is one. Farther away, it approaches zero. A narrow width makes the unit respond strongly only near its centre. A wide width allows more distant inputs to produce a substantial response.
The unit divides the squared distance by twice the square of its width before turning it into a response, so the width sets the scale on which distance is judged. One unit of distance is far for a width of one half and near for a width of three. The playground’s three units, read at an input of one, show it.
| Width of each unit | Unit centred at −1 | Unit centred at 0 | Unit centred at 1 |
|---|---|---|---|
| 0.5 | 0.0003 | 0.1353 | 1.0000 |
| 1 | 0.1353 | 0.6065 | 1.0000 |
| 2 | 0.6065 | 0.8825 | 1.0000 |
| 3 | 0.8007 | 0.9460 | 1.0000 |
The three responses at an input of one, as the playground’s table reports them at four settings of the width.
The unit sitting on the input answers one at every width. At a width of one half the unit two away has all but stopped answering. At a width of three it still answers 0.8007, and the three responses have become hard to tell apart.
These responses are features for another layer. They do not automatically represent probabilities, and their sum across units does not have to equal one. At a width of one the three responses above add to about 1.74.
One input, three local responses
The default input is one. The centres are minus one, zero, and one, and all three widths are one. Twice the squared width is therefore two, which is the divisor in the second calculation.
Squared distances = [(1 − (−1))², (1 − 0)², (1 − 1)²]
= [4, 1, 0]
Responses = [exp(−4 / 2), exp(−1 / 2), exp(0)]
≈ [0.1353, 0.6065, 1.0000]The input exactly matches the third centre, so that unit gives its maximum response. The other two still contribute information about the input's location relative to their regions.
Now move the input to one half, midway between the second centre and the third.
Squared distances = [(0.5 − (−1))², (0.5 − 0)², (0.5 − 1)²]
= [2.25, 0.25, 0.25]
Responses = [exp(−2.25 / 2), exp(−0.25 / 2), exp(−0.25 / 2)]
≈ [0.3247, 0.8825, 0.8825]The second and third units give the same response. The input is half a unit above one centre and half a unit below the other, and a squared distance cannot tell those two apart. No unit answers one now, because the input sits on no centre.
Part 2. Calculating and Learning the Response
3. Write the Gaussian unit
For input vector x, centre c, and positive width w, define the squared distance s and output o:
s = Σ_k (x_k − c_k)²
o = exp(−s / (2w²))Each piece has one job. The squared distance s is the measurement from Part 1. Dividing it by 2w² judges it against the width. The minus sign makes a larger distance give a smaller exponent, and the exponential turns that exponent into a number between zero and one, which is one exactly when s is zero.
Both the centre and width can be learned. The SDK stores one centre vector and one width per unit. It records squared distances in the forward response so the backward pass can use the same values.
4. Find how each parameter affects the output
Let g be the arriving derivative of the loss with respect to one unit's output. Applying the chain rule gives:
∂L/∂c_k = g o (x_k − c_k) / w²
∂L/∂w = g o s / w³
∂L/∂x_k = −g o (x_k − c_k) / w²›Where the three derivatives come from
Write the exponent as u, so the output is the exponential of u and the derivative of the output with respect to u is the output itself. What remains is how u changes with a centre coordinate, an input coordinate and the width.
u = −s / (2w²), o = exp(u), ∂o/∂u = o
∂u/∂c_k = (x_k − c_k) / w² because ∂s/∂c_k = −2 (x_k − c_k)
∂u/∂x_k = −(x_k − c_k) / w² because ∂s/∂x_k = 2 (x_k − c_k)
∂u/∂w = s / w³ because ∂(w⁻²)/∂w = −2 w⁻³Multiplying each line by the output and by the arriving derivative g gives the three results above. The width is the only parameter that ends up cubed, and it does so because it was already squared inside the exponent.
The playground’s derivatives at the default input
Open the parameter derivatives in the playground. It sends a derivative of one into every unit, so g is one for all three, and what the table reports is the derivative of the sum of the three responses. The input is one and every width is one.
Centre derivatives, o × (x − c) / w²
unit centred at −1 0.1353 × (1 − (−1)) / 1² ≈ 0.2707
unit centred at 0 0.6065 × (1 − 0) / 1² ≈ 0.6065
unit centred at 1 1.0000 × (1 − 1) / 1² = 0
Width derivatives, o × s / w³
unit centred at −1 0.1353 × 4 / 1³ ≈ 0.5413
unit centred at 0 0.6065 × 1 / 1³ ≈ 0.6065
unit centred at 1 1.0000 × 0 / 1³ = 0Every centre derivative is positive or zero, because the input lies above the first two centres and moving a centre up, toward the input, raises its response. The unit already on the input has nothing to gain from moving, and nothing to gain from a different width either.
The nearer of the other two units has the larger centre derivative, 0.6065 against 0.2707. The far unit is displaced twice as much, but its response is less than a quarter of the near unit’s, and the derivative is the product of the two. A unit that barely responds to an input learns little from it.
With every width at one, w² and w³ are both one, so this example cannot show which power of the width each formula carries. Any other width can, and the practice problems at the end run two of them.
The centre and input derivatives have opposite signs because moving either one toward the other reduces their distance. Increasing width raises the response away from the centre. At the centre itself, changing width has no effect on the peak value.
A batch adds parameter contributions across observations. Each input receives the sum of the derivatives from all the units that read it. Training then subtracts a learning-rate-scaled gradient from each parameter.
Part 3. Combine Local Responses into a Prediction
A radial basis network can place a dense output layer after these units. The local responses tell the output layer which regions match the input, and the output weights determine how those matches contribute to a prediction.
A prediction from three responses
Suppose the output layer holds one weight for each unit, 2, −1 and 3, and no bias. These weights are chosen for the illustration and have not been trained. The prediction is each response times its weight, added up, with the responses taken at full precision.
At an input of 1, the responses are about [0.1353, 0.6065, 1.0000]
prediction = (2 × 0.1353) + (−1 × 0.6065) + (3 × 1.0000) ≈ 2.6641
At an input of 3, the responses are about [0.0003, 0.0111, 0.1353]
prediction = (2 × 0.0003) + (−1 × 0.0111) + (3 × 0.1353) ≈ 0.3956At an input of one the third unit answers one, so its weight of 3 supplies most of the prediction. At an input of three, two units of distance from the nearest centre, every response is small, and the prediction is small whatever the weights are. Far from all the centres it settles at the output layer’s bias, which is zero here. A radial basis network describes the regions around its centres and says very little about anywhere else.
This makes feature scale consequential. If one coordinate is measured in thousands and another in tenths, squared Euclidean distance can be dominated by the first. Choose meaningful units or fit feature scaling on the training data before fitting the model.
Keep in mind
Very narrow units can respond to almost none of the observed data, producing weak learning signals away from their centres. Very wide units can become difficult to distinguish. Widths must stay positive; the SDK refuses a training step that would make one zero or negative.
The playground reaches both ends. At a width of 0.2 and an input of three, all three responses are zero to four decimal places and so are all six parameter derivatives, so that input would teach these units nothing. At a width of three and an input of one the responses are 0.8007, 0.9460 and 1.0000, three numbers that tell the output layer almost the same thing.
The playground displays derivatives of the sum of the three responses so their directions can be inspected. A trained predictor supplies its own loss derivatives, which can have different signs and magnitudes.
Check yourself
Questions on Parts 1 to 3
Question 1 of 5
The input is one, the centres are minus one, zero and one, and all three widths are one. What does the third unit answer, and why?
Try it with Botie
Build with Botie
Write and run Python