Outages we hear about from monitoring, not customers.

The Prometheus metrics had been green during each of the three outages. Components were healthy. The problem was an interaction — the API was up, the database was up, but a flag-gating service was misbehaving and the combination broke checkout. No single component alarm had fired.

We wrote CloudWatch Synthetics canaries — actual Node.js scripts that drove a headless browser through fourteen critical user journeys: login, search, checkout, settings update, the high-value workflows. Each ran every minute from three regions. Alerts routed through EventBridge to PagerDuty.

Mean time to detection on customer-journey breakage dropped from 38 minutes to under 2 minutes. In the 90 days post-rollout, customers never reported a problem before monitoring did. False-positive rate stayed at about 2 per month — well-tuned against the noise floor.