Build a Card-Fraud Detector

• ✓Show why a model that is 99.8% accurate and catches no fraud is worthless, and score rare events with precision-recall instead
• ✓Put a bootstrap interval on a metric measured from only a few dozen positives
• ✓Split transactions by time and explain why a shuffled split and early resampling both leak
• ✓Compare logistic regression, a random forest and gradient boosting fairly, over several seeds
• ✓Choose an alert threshold from what a missed fraud and a false alarm each cost
• ✓Check and repair probability calibration after class weighting, and report precision at the alert volume analysts can review
• ✓Write a model card that states what the model cannot see and when to stop trusting it

Do what a fraud analyst's team does with a new model — judge it on the right metric, test it on the future, set the alert threshold in money, and say how many alerts the team can review.