Anomaly Detection

Thinking Tools6.4 Conducting Follow-Up and Taking Feedback

From statistical process control and operations research, descended from the control charts Walter A. Shewhart drew at Bell Labs (1924; Economic Control of Quality of Manufactured Product, 1931), and surveyed for modern data by Varun Chandola, Arindam Banerjee, and Vipin Kumar (Anomaly Detection: A Survey, 2009).

Makes three choices explicit for any watched measure, the normal baseline, the threshold for different enough, and who acts when it fires, producing early warning of real shifts.

An alert on daily refund requests: where the threshold sits decides false alarms against missed problems

On a normal day an online store sees about 40 refund requests, give or take 6; a real problem, such as a bad batch or a fraud ring, pushes the count up. The bars show where nineteen in twenty days of each kind land, and the shaded overlap is where the two cannot be told apart. Move the alert threshold through the overlap and watch the two kinds of error trade against each other; then shrink the shift a real problem causes. Illustrative.

50
18
False alarms a year, on normal days
20
Real problem days missed, of every 10
0.9
Either kind of dayNormal days: 28 to 52 refundsDays with a real problem20406080Alert thresholdEither kind of dayNormal days: 28 to 52 refundsDays with a real problem20406080Alert threshold

Either kind of dayFrom 46 to 52No overlap: a threshold between the bars catches every real problem with no false alarms.

An example

An online home-goods store found out about a fraud ring from its bank after three weeks of refunds to stolen cards. The daily refund count had been on a dashboard the whole time. Nobody owned it.

The operations lead rebuilt the watch as three explicit choices. The baseline: 180 days of refund requests, about 40 a day with a spread of 6. The threshold: an alert above 50 a day, chosen after counting the cost of each kind of error. At 50, the team would see about twenty false alarms a year, each costing half an hour of an analyst's time, and would miss about one real problem day in ten; at 56 the false alarms fell to four but the misses rose to one in three. A missed fraud day cost far more than twenty half-hours, so 50 stood. The owner: the payments analyst on duty, with a checklist and the authority to pause refunds to new cards. Each alert was labeled real or false afterward, and the threshold was reviewed every quarter.

Why it works

Every measure moves, and most movement is noise. Watching a number without rules produces one of two failures: people react to every wiggle and burn out, or they learn to ignore the number and miss the real shift. The tool replaces the vague watch with three decisions written down: what normal looks like, how far from normal is far enough to act, and who acts.

The threshold is where the judgment sits, and it is a trade. Normal days and problem days overlap, so no threshold catches every real problem without false alarms. Moving it lower catches more problems and raises more false alarms; moving it higher does the reverse. Pricing each kind of error makes the choice a business decision instead of a statistical default, and labeling each alert afterward shows whether the trade is working.

The thinking behind it

Walter A. Shewhart Walter A. Shewhart, a physicist and statistician at Bell Telephone Laboratories, sketched the control chart in a 1924 memo and set out the method in Economic Control of Quality of Manufactured Product (1931). His central distinction was between common-cause variation, the noise any process produces, and assignable causes that deserve investigation. Limits drawn around a stable baseline tell an operator when to act and, as important, when to leave the process alone.

Varun Chandola Varun Chandola, with Arindam Banerjee and Vipin Kumar, wrote Anomaly Detection: A Survey (2009), the standard map of the field's methods for modern data. It organizes the techniques, among them classification, nearest neighbor, clustering, and statistical approaches, by the kind of anomaly and the kind of data they handle, which helps a team pick a method that matches its signal before it worries about thresholds.

How to run it

  1. Define the signal. Name the measure, how often it is sampled, and what a real problem would do to it.
  2. Build a documented baseline. Use enough history to describe normal, split by known patterns such as weekends, seasons, and promotions.
  3. Price the two errors. Estimate what a false alarm costs in time and attention and what a missed problem costs in money or harm.
  4. Set the threshold from those prices. Move it until the expected cost of misses and false alarms together is lowest, and write down why.
  5. Name the responder. Assign one owner per alert, with a checklist and the authority to act, such as pausing a process.
  6. Label every alert afterward. Record whether each was real or false, and whether any problem was found that the alert missed.
  7. Revalidate on a fixed cadence. Review the baseline and threshold quarterly, or after a known change in the business.

Where it goes wrong

The common failure is alert fatigue. A threshold set too low fires daily, responders learn to dismiss it, and the real alert is dismissed with the rest. It shows up as alerts closed in seconds with no note. Raise the threshold, or add a second condition, until most alerts are worth opening.

A second failure is a stale baseline. The business grows or changes its returns policy, normal shifts, and the old threshold either fires constantly or never. Rebuild the baseline after every known change. A third is the orphan alert: a well-tuned rule that sends email to a shared inbox nobody owns. Every alert needs one named person and a defined first action.

When to use something else

  • Control Chart. When the measure comes from a repeatable process and the job is full control charts with rules for runs and trends.
  • Regression to the Mean. When an extreme reading has already been acted on and the question is whether the next one would have come back anyway.
  • Goodhart-Aware Metric Selection. When the worry is that the watched measure is being gamed, so a clean signal is not a true one.

Where it sits in SOLVE eX

Go to the source

  • Control chart memorandum, Bell Laboratories, 1924
  • Economic Control of Quality of Manufactured Product, 1931
  • Anomaly Detection: A Survey, ACM Computing Surveys, 2009

Back to the catalog