Logistic Regression

Each dot in the box above is one student. The input x is the number of hours they studied, and the outcome y is whether they passed. Fails sit on the floor of the plot and passes on the ceiling, because the outcome is one or the other with nothing between.

y = 1 for passed        y = 0 for failed

Zero and one are labels, and nothing more. Passing is not one unit more than failing, and the model will not treat it so. The twelve students the page works with have a mixed middle. Below three and a half hours everyone failed, above five and a half everyone passed, and between them the two outcomes are tangled together. Someone at four and a half hours might go either way, and the model is going to use the hours to estimate how likely each way is.

The obvious first attempt is the regression page’s line, fitted to the zeros and ones as if they were numbers.

A straight line can produce any value. A probability has to stay between zero and one. This line predicts −0.09 for a student who studied one hour and 1.18 for one who studied eight, and neither is a chance of anything. The red stretches are every input at which the line has stopped meaning what it was asked to mean. A probability model has to return a value between zero and one for every possible input, not only for the inputs it happened to see.

Three quantities run through this page, and keeping them separate is most of understanding the model.

They are related, and each is computed from the one before, and they are not the same thing. The score is what the straight line produces. The probability is what the score becomes. The decision is a rule laid on top of the probability, and it is the only one of the three that has a threshold in it.

Start with the calculation every linear model starts with.

z = β·x + α

The score can be any real number. Positive scores favour passing, negative scores favour failing, and zero is neutral. The score itself is not a probability. The fitted slope is about 1.69 and the intercept about −7.18.

score after four study hours = 1.69 × 4 − 7.18 = −0.42
score after five study hours = 1.69 × 5 − 7.18 = 1.27

These calculations use rounded coefficients. The full fitted values give approximately −0.40 and 1.29. The sigmoid will turn each score into a probability.

Whatever turns a score into a probability should send large negative scores near zero, send large positive scores near one, send zero to exactly one half, and do all of that smoothly. The sigmoid does.

p = σ(z) = 1 / (1 + e⁻ᶻ)

The sigmoid converts an unrestricted score into a value between zero and one, and it never quite reaches either end. A finite score always gives a probability strictly inside the band.

The amber steps on the sigmoid above are three moves of exactly one unit along the score axis, at −4, at 0 and at 4. Read what each does to the probability.

The same step in the score buys a small change near zero, the largest change near one half, and a small change again near one. The model is linear in its score, and its probability does not move by a constant amount. That is not a flaw to be fixed. A model that has already assigned a 98% chance has less room to become more convinced, and the curve says so.

A probability of 0.75 is three passes in every four students. Said the other way, it is three passes for every one fail, and that is the odds, three to one.

odds = p / (1 − p)

Probability compares the passes with everyone. Odds compare the passes with the fails. They carry the same uncertainty in a different comparison, and the tokens in the widget below are the two comparisons side by side.

Odds run from zero up to infinity, and a straight-line score runs from minus infinity to plus infinity, so the two still do not match. Take the logarithm of the odds and they do.

log-odds = ln( p / (1 − p) )

A probability below one half has odds below one and negative log-odds. A probability of exactly one half has odds of one and log-odds of zero. A probability above one half has odds above one and positive log-odds. Three scales, one centre.

Now the defining relationship, which section 4’s score and section 8’s log-odds turn out to be the same number.

ln( p / (1 − p) ) = β·x + α

Logistic regression models the log-odds with a straight line. The sigmoid is that relationship solved for p, so the two forms are one statement read in two directions.

p = σ(β·x + α)

This is the bridge the page is built on. The model is not linear in the probability, which section 6 showed, and it is not linear in the odds. It is linear in the log-odds, and every reading of a coefficient in the next Part is a reading on that scale.

At zero hours the score is α on its own.

z(0) = α          p(0) = σ(α) = σ(−7.18) = 0.0008

The intercept is the log-odds when the input is zero, and the sigmoid of it is the probability there. For these students that is a chance of passing of eight in ten thousand with no study at all, which is the fitted model’s baseline. Its practical meaning is only as good as zero is as an input.

Nobody here studied zero hours, so the number is the line’s position more than a claim about anyone, exactly as the regression page’s intercept of −68 kg was. Centring the hours would make the intercept the log-odds at the average study time instead.

For every additional hour the score rises by β, which is to say the log-odds rise by β, which is to say the odds are multiplied by e^β.

β = 1.69          e^β = 5.44

Under the fitted model each additional hour multiplies the odds of passing by about 5.4. The widget below picks a starting time and follows one more hour through all three scales.

The two sliders in the widget also show what each setting controls. Moving α slides the whole transition left or right, changing the probability at every study time and where the curve crosses one half. Moving β changes how sharp the transition is and, through zero, which way it runs. A positive β rises with x, a negative one falls, and a zero gives a flat line at σ(α).

dp/dx = βp(1 − p)
at p = 0.5: dp/dx = β × 0.5 × 0.5 = β/4
with β ≈ 1.69: dp/dx ≈ 1.69 / 4 ≈ 0.42 per hour

The model can report a probability and stop there, and often that is the right thing to report. When a yes or a no is needed, a threshold t supplies it.

ŷ = 1 when p ≥ t          ŷ = 0 when p < t

At t = 0.5 the score threshold is zero, and the boundary on the hours axis is where the line crosses zero. For any other threshold the boundary is where the log-odds reach the threshold’s own log-odds.

boundary x = ( logit(t) − α ) / β          logit(t) = ln( t / (1 − t) )

at t = 0.5:  logit(0.5) = 0, so x = −α / β = 7.18 / 1.69 = 4.24 hours

The model estimates a probability. A separate threshold converts that probability into a class, and 0.5 is the conventional choice rather than a mandatory one.

Slide the threshold below. The curve does not move, and nothing is refitted. What moves is the horizontal line, the boundary on the hours axis, and which students land on which side of it.

At 0.5 the model gets ten of twelve right, missing the student who failed after five hours and the one who passed after four. Lower the threshold to 0.3 and the boundary moves to 3.74 hours, the four-hour pass is caught, and accuracy rises to 0.917 with nothing about the curve changed. Raise it to 0.8 and the boundary moves to 5.06 hours, the five-hour fail is caught instead, and two passes are missed. Changing the decision threshold changes the classifications without changing the fitted model.

Whether 0.3 or 0.8 is better than 0.5 is not a question the model can answer, because it depends on what the two kinds of mistake cost. Suppose the point of the prediction is to offer help to students at risk of failing. Missing a student who needed help is expensive. Offering help to one who did not is cheap. Then the threshold for calling someone a pass should be high, so that more borderline students are flagged as possible fails, at the price of more false alarms.

A useful threshold depends on the consequences of the different mistakes, and choosing it properly belongs to the judging-a-classifier page. What this page needs is the fact that the threshold is a policy decision sitting on top of the probabilities, and not part of the model.

The four cells under the threshold widget separate the four things that can happen to a student.

accuracy = correct predictions / all predictions

Click a cell and the students it counts light up. Two things about accuracy are easy to miss. It evaluates thresholded decisions and says nothing about the probabilities that produced them. And it compresses four kinds of outcome into one number, so a model that misses every rare case can still score well. The section 13 numbers make the first point. Accuracy went from 0.833 to 0.917 while the log loss stayed at 0.2933, because the probabilities never changed.

Take a student who passed and two predictions for them, p = 0.9 and p = 0.6. Both become a pass at a threshold of 0.5, so accuracy cannot tell them apart. The first assigned more probability to what actually happened, and it should count as the better prediction. A student who failed, predicted at 0.1 or at 0.4, is the same story from the other side.

Fitting a probability model needs a score that separates predictions which make the same decision at different confidence. Accuracy is not it, and the next three sections build the one that is.

For one student, ask how much probability the model gave to the outcome that occurred. If they passed that is p, and if they failed it is 1 − p.

P(yᵢ | pᵢ) = pᵢ        when yᵢ = 1
P(yᵢ | pᵢ) = 1 − pᵢ    when yᵢ = 0

P(yᵢ | pᵢ) = pᵢ^yᵢ (1 − pᵢ)^(1 − yᵢ)     both cases in one line

The exponent trick is worth reading once. When yᵢ = 1 the second factor has exponent zero and vanishes, and when yᵢ = 0 the first does. Multiply over every student and the result is the likelihood, the probability the model assigned to the complete set of outcomes that happened.

L = ∏ pᵢ^yᵢ (1 − pᵢ)^(1 − yᵢ)

For the fitted curve on the twelve students the product is 0.0296. That is small, and it should be, since it is the chance of twelve specific things all happening. What matters is that it is larger than any other curve makes it.

Products are awkward to differentiate and quickly become numbers too small to store, so take the logarithm. Products become sums, and the sums are easy.

ℓ = Σ [ yᵢ ln(pᵢ) + (1 − yᵢ) ln(1 − pᵢ) ]

Turn it negative and average it, and it becomes a loss to be made small, called the binary log loss.

log loss = −ℓ / n

Per student the loss is −ln(p) for a pass and −ln(1 − p) for a fail, and the two curves say what confidence costs.

The cost climbs without limit as the curve grows confident in the wrong outcome. In the fitted table the two biggest costs are the two students in the mixed middle, 1.536 for the five-hour fail the curve put at 0.785, and 0.914 for the four-hour pass it put at 0.401. Log loss strongly penalises a confident prediction that assigned little probability to what actually happened.

The second curve in the table above has half the fitted slope and half the fitted intercept. It crosses one half at the same 4.24 hours, so it makes exactly the same twelve decisions and has exactly the same accuracy of 0.833. Its log loss is 0.3425 against the fitted curve’s 0.2933, because it is less sure about the students it is right about, and being less sure costs.

Logistic regression is fitted to probability quality. It maximises the likelihood, and it does not directly choose the boundary that produces the fewest classification mistakes. On these students the two happen to agree. They need not, and section 13 already showed a threshold that beats the fitted curve’s default accuracy without touching its probabilities.

Two parameters, α and β. Every pair gives twelve scores, twelve probabilities and one log loss, so the loss is a surface over the pair, and fitting is the fitting-by-walking page’s search across it. The surface below is the log loss over intercept and slope, and the path is the climb itself, recorded pass by pass.

The walk starts with both coefficients at zero. That assigns a probability of one half to every student, regardless of study hours.

initial log loss = −ln(0.5) = ln 2 ≈ 0.693

The optimum on this dataset is near intercept −7.18 and slope 1.69, with loss 0.293. At learning rate 0.5, the first update overshoots and the loss rises to 0.743 before falling. It settles after a few thousand passes. Fitting means finding coefficients whose probability predictions minimize the loss.

A pass here has the same shape as on the walking page with one extra stage, the squash. Begin with the current α and β. Compute each score zᵢ. Convert each to pᵢ. Compute each student’s loss and average. Compute the gradient. Update both coefficients. Draw the new curve.

Scrub the widget above one pass at a time from the start and watch the curve panel. The first few passes lurch, because the rate is bold, and after that the curve steepens steadily into place. Logistic regression repeatedly turns inputs into probabilities, measures their loss, and adjusts its coefficients, and nothing else.

The gradient of the log-likelihood has a form worth remembering, because it looks like something already seen.

∂ℓ/∂α = Σ (yᵢ − pᵢ)
∂ℓ/∂β = Σ (yᵢ − pᵢ)·xᵢ

The term yᵢ − pᵢ is the gap between what happened and what the curve expected, a residual on a probability scale. At a finite, unpenalised optimum both sums are zero.

Σ (yᵢ − pᵢ) = 0          Σ (yᵢ − pᵢ)·xᵢ = 0

The first says the total of the predicted probabilities equals the number of passes, so the curve is right on average. The second says the gaps also balance after weighting by hours, so the curve is right on average at the low end and the high end separately. The bars below are the gaps on the fitted curve, student by student.

Scrub the passes and the two totals under the bars, which are the two gradient components, shrink toward zero as the climb settles. The regression page’s residuals summed to zero for the same reason, and the gap here is the residual with a probability standing where the prediction stood.

The derivation needs the sigmoid’s own derivative first, and it is the tidiest in the subject.

The sigmoid changes fastest when its output is one half. Substitute that output into the derivative to see the maximum slope.

Maximum sigmoid slope = 0.5 × (1 − 0.5) = 0.25

The slope approaches zero as the output approaches either extreme. This explains why a saturated sigmoid responds so little to a small change in its input score.

Take one student and follow β through the chain to their contribution to the log-likelihood.

ℓᵢ = yᵢ ln(pᵢ) + (1 − yᵢ) ln(1 − pᵢ)
β → zᵢ → pᵢ → ℓᵢ

The messy fractions from the logarithms and the p(1 − p) from the sigmoid cancel each other exactly, and what survives is the gap times the input. Only now sum over the students, and the compact gradient of section 23 is the total of these per-student pieces.

Set the two gradients to zero and compare with the regression page. There, the prediction was βxᵢ + α, the conditions were linear in β and α, and algebra solved them. Here pᵢ = σ(βxᵢ + α) wraps the coefficients inside an exponential, and the conditions

Σ (yᵢ − σ(βxᵢ + α)) = 0          Σ (yᵢ − σ(βxᵢ + α))·xᵢ = 0

cannot in general be rearranged into a formula for β and α. The equations exist, and they are the right ones. They do not have a closed-form solution, and that is why the model is fitted by walking. The fit here climbs the log-likelihood by gradient ascent, which is the same thing as descending the log loss, and reports how many passes it took and whether it met its tolerance.

Now twelve students with no mixed middle. Every fail is at three and a half hours or less and every pass at five and a half or more, so a boundary anywhere between them classifies everyone perfectly.

Scrub the passes and read the three traces together. Accuracy reaches one by pass 100 and never changes again. The slope keeps growing, 1.05 at pass 100, 2.32 at pass 1000, and 3.50 at pass 5000 if the budget allows it. The log loss keeps falling, 0.133, 0.024, 0.006. Every steepening of the curve makes the observed outcomes more likely, since there is no student in the middle to be wrong about, and the climb never finds a top.

The repair is the ridge page’s. Add a penalty on the size of the coefficients to the loss, and the climb gains a competing cost. Steepening the curve still raises the likelihood and now also raises the penalty, and at some finite slope the two pulls balance. Penalised logistic regression settles on separated data at a finite, less extreme curve, and the probabilities it reports stay away from zero and one.

The model estimates a conditional probability.

P(y = 1 | x)

For the students that is the estimated chance of passing, given the hours supplied and the fitted model. It does not mean the student is partly passed, or that the model knows the true chance, or that study time is the only thing that matters, or that study time causes the outcome, or that two students at the same hours are in the same circumstances. It is a model-based estimate conditioned on the one feature it was given.

There is also a test the number should pass. If a model assigns many comparable students a probability near 0.8, then about 80 percent of them should turn out to pass. That agreement between stated confidence and observed frequency is calibration, and a model can classify well while being poorly calibrated.

Reading a calibration table takes three steps. Sort the students by the probability the curve gave them and cut that range into bins. Inside each bin, average the predicted probabilities, which is what the curve claimed on average for those students. Then count how many of them actually passed, which is what happened. A calibrated curve has the two columns agree in every bin that holds enough students for a rate to mean anything.

The lowest bin holds the five students who studied three and a half hours or less. The curve put them at 0.074 on average and none of them passed, so claim and outcome agree. The highest bin holds the five who studied five hours or more. The curve put them at 0.924 on average and four of the five passed, the five-hour fail being the exception, so the curve claimed a little more there than it delivered. The two middle bins are the four-hour pass at 0.401 and the four-and-a-half-hour pass at 0.610, one student each, and a pass rate over one student is not a rate.

So far the score has read one number, the hours. Suppose each student were measured twice, hours studied and, say, hours slept. Each input gets a coefficient of its own, and the score is each input times its coefficient, added up, plus the intercept. That weighted sum is the multiple regression page’s dot product, and everything after it is unchanged.

z = β₁x₁ + β₂x₂ + α          p = σ(z)

written for any number of inputs:  z = β·x + α

The sigmoid turns the score into a probability exactly as before, the log loss scores that probability against what happened, and the fit is the same walk with one more direction to move in. The gradient gains one component per input, and each component is the gap between outcome and probability times that input, summed over the students. It is the single-input gradient of Part 7 with the second input standing where the hours stood.

∂ℓ/∂βⱼ = Σ (yᵢ − pᵢ)·xᵢⱼ          one such sum for each input j

Each coefficient changes the log-odds while the other inputs are held fixed, so one more hour of study multiplies the odds by e to the first coefficient among students who slept the same amount, with the same qualification about held-fixed comparisons the multiple regression page gave.

The threshold of one half falls where z = 0, and with two features that is a line across the plane of the two inputs. Every student on one side of it is called a pass and every student on the other a fail, however far from the line they sit. With three it is a plane, and with more it is a flat surface in more dimensions than can be drawn.

boundary at t = 0.5:  β₁x₁ + β₂x₂ + α = 0

Adding features changes the geometry of the boundary and nothing about the mechanism. The multiclass page takes the next step, from two classes to many.