Glossary

The words chapters and interviewers use, each in a sentence or two. Tap a term in any chapter to see the same definition.

AUC: Area under the ROC curve: the chance that a randomly chosen positive gets a higher score than a randomly chosen negative. 0.5 is a coin flip and 1 is a perfect ranking; it says nothing about where to put the threshold.
Base rate: How common the positive class is, such as 1 urgent ticket in 100. A low base rate drags precision down even for a model that ranks well.
Break-even point: The score at which two choices have the same expected cost. With calibrated scores it equals the cost of a false positive divided by the sum of both mistakes' costs.
Calibration: A model is calibrated when its scores can be read as probabilities: of the cases scored around 0.07, about 7 in 100 are positive. A model can rank well and still be badly calibrated.
Confusion matrix: A two-by-two count of a classifier's outcomes at one threshold: true positives, false positives, false negatives and true negatives. Most classifier metrics are ratios of these four counts.
Control group: Another name for a holdout: the randomly chosen people who don't get the action in an experiment.
Effect: The difference an action makes: the outcome with the action minus the outcome without it, usually measured against a randomly chosen holdout.
Expected cost: What a choice costs on average: each outcome's cost times its probability, added up. Comparing the expected cost of each choice sets a cost-based threshold.
F1: The harmonic mean of precision and recall. It treats a false positive and a false negative as equally bad, so it is a weak guide when the two mistakes cost different amounts.
False negative: A case the model misses that is what it looks for, such as a defaulter approved or an urgent ticket left in the queue.
False positive: A case the model flags that turns out not to be what it looks for, such as a good loan applicant declined or a needless page.
Holdout: A group picked at random and deliberately not acted on, so its results show what would have happened without the action.
Lead time: How far ahead of an event a prediction arrives. A prediction is only useful if it arrives while an action can still change the outcome.
Precision: Of the cases a model flags, the share that really are positive. Raising the threshold usually raises precision and lowers recall.
Precision-recall curve: A plot of precision against recall, one point per threshold. When positives are rare it shows the false alarms that the ROC curve hides.
Recall: Of all the real positives, the share the model flags. Also called the true positive rate or sensitivity.
ROC curve: A plot of the true positive rate against the false positive rate, one point per threshold. It shows how well a model ranks, whatever threshold is used.
Threshold: The rule that turns a model's score into an action, such as 'decline above 0.09'. The model produces the score; the business chooses the line.
Uplift: The effect of an action on a particular kind of person or account. The highest-risk cases aren't always the ones an action helps most, so aiming at high uplift can beat aiming at high risk.