Overfitting
Description
Overfitting is a fit driven so close to the specific sample it was derived from that it absorbs the sample’s idiosyncratic noise as if it were signal. Any finite sample contains both the generalizable structure it shares with the wider population (the signal) and accidental, sample-specific variation (the noise). A fitting process with enough flexibility will reduce error against the sample by modeling the noise too — and every bit of noise it models is a commitment that will not hold for new cases. The signature is a widening gap between in-sample performance (which keeps improving) and out-of-sample performance (which eventually degrades). The concept picks out a precise shape: high fit to the observed cases AND poor transfer to unobserved cases from the same source AND the in-sample view failing to disclose the problem. That last clause matters — the overfit model looks excellent by the only evidence in front of it, which is why the failure needs held-out data, cross-validation, or a regularizing pressure to surface. The counter-moves all trade a little in-sample fit for transferability: simpler models, regularization penalties, more independent data, early stopping. Its home is statistics and machine learning, but the structural shape recurs anywhere a pattern is inferred from limited experience and then relied on for new situations.Triggers
User-initiated: User describes a model, rule, or lesson that performs well on the cases it was built from but breaks on new ones, or worries a fit is “too good.” Vocabulary cues: “overfit,” “fails to generalize,” “memorized the training data,” “works in backtest, not live,” “learned the exceptions.” Agent-initiated: Agent notices a pattern being extracted from a limited sample and applied to new cases without a transfer check. Candidate inference: “is this capturing the signal shared with the wider population, or the sample’s own noise? What would held-out data say?” Situation-shape signals: In-sample metrics that keep improving while held-out metrics turn; rules elaborated to cover every past case; strategies tuned on historical data; any generalization drawn from a single run of experience.Exclusions
- Cargo-cult imitation — cargo-cult copies a surface shape with no underlying force at all. Overfitting is the opposite failure: it captures real, load-bearing structure that genuinely fits the sample, but that structure is local to the sample and does not transfer. The overfit fit did work on its data; that is precisely the trap.
- Local-minimum — a local-minimum is a position in an optimization landscape (stuck, every nearby step looks worse). Overfitting is a gap between in-sample and out-of-sample performance. One names where you are on the landscape; the other names whether your fit generalizes off it.
- Underfitting or an appropriately matched fit — when the model is too simple to capture the signal (underfit) or exactly matched to it (a good fit that generalizes), overfitting is the wrong diagnosis. The failure, if any, is on the other side of the bias-variance trade-off, or there is no failure.
- Genuine distribution shift — when transfer fails because the world changed (the new cases are drawn from a genuinely different distribution than the sample), the fit did not mistake noise for signal; it captured real structure that no longer holds. That is covariate or concept drift, a distinct failure mode.
Structure
Relationships
- goodharts-law — the analogical twin: a proxy pushed until it no longer stands for its target. Overfitting’s proxy is the sample standing for the population; Goodhart’s is the metric standing for the goal.
- cargo-cult — the sharp contrast: cargo-cult reproduces surface with no force; overfitting reproduces a genuine but non-transferable fit. Distinguishing them separates “never worked” from “worked only here.”
- trade-off — overfitting sits at the high-variance end of the bias-variance trade-off; naming the trade-off names the dial whose over-turning produces it.
Examples
Fitting a high-degree polynomial to noisy data · computer-science
Fitting a high-degree polynomial to noisy data · computer-science
A high-capacity model driven to zero error threads through every training point exactly — including the measurement noise — and then oscillates wildly between them, predicting new points from the same source badly. The fit is perfect on what it has seen and useless on what it has not; the giveaway is that adding held-out points, or penalizing the model’s flexibility, sharply improves it.
Skinner, B. F. (1948). "'Superstition' in the Pigeon." Journal of Experimental Psychology, 38(2), 168-172. · psychology
Skinner, B. F. (1948). "'Superstition' in the Pigeon." Journal of Experimental Psychology, 38(2), 168-172. · psychology
Skinner delivered food to hungry pigeons on a fixed time interval, entirely independent of anything the birds did. The pigeons nonetheless developed idiosyncratic “superstitious” rituals — one turning counter-clockwise, another bobbing its head, another swinging toward a corner — each bird having repeated whatever it happened to be doing when food last arrived, as if that action had produced it.Inference: each pigeon overfit its own sample of experience. The coincidence of a behavior and a reward is noise (the food was time-based, unrelated to behavior), but a learner flexible enough to bind any co-occurring action to the reward will fit that noise as if it were a causal signal. The corrective is the same as in statistics: more independent trials reveal that the contingency does not hold, that the ritual is local to the moments it was learned from and transfers to nothing. It is also the clean line against cargo-cult — the pigeon captured a real local co-occurrence, not an empty copied form; the structure was there, just spurious.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. · statistics
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. · statistics
The standard treatment frames overfitting through the bias-variance decomposition of expected prediction error. As model complexity increases, training error falls monotonically toward zero, but test error follows a U-shape: it declines while added flexibility captures signal, then rises as the flexibility begins fitting the training sample’s noise. The gap between the two curves is the generalization gap, and the minimum of the test-error curve is the complexity the sample can actually support.Inference: this is why in-sample fit cannot be trusted as evidence of a good model — it improves under exactly the flexibility that produces overfitting. The remedy is structural rather than exhortatory: hold out data (validation sets, cross-validation) so the test curve is observable, and apply regularization to penalize the flexibility that buys training fit at the cost of transfer. The book makes overfitting a property of the complexity-versus-sample-size relationship, not a mistake of any one model.