AWS US-EAST-1 DynamoDB DNS outage
A roughly 15-hour outage in Amazon's largest cloud region (US-EAST-1) took down or degraded over 100 AWS services and rippled out to major platforms including Snapchat, Fortnite, Roblox, Signal, Coinbase and Venmo, among many others.
The damage was public. The root cause was preventable.
A roughly 15-hour outage in Amazon's largest cloud region (US-EAST-1) took down or degraded over 100 AWS services and rippled out to major platforms including Snapchat, Fortnite, Roblox, Signal, Coinbase and Venmo, among many others.
What the software actually got wrong
AWS traced the root cause to a race condition in the automated system that manages DNS records for the DynamoDB database service's regional endpoint, which caused that endpoint's DNS record to be emptied out; every service that needed a fresh connection to DynamoDB in that region began failing DNS resolution, and the failure then cascaded into EC2's instance-launch subsystem and Network Load Balancer health checks, prolonging the recovery.
AWS traced the root cause to a race condition...
AWS traced the root cause to a race condition in the automated system that manages DNS records for the DynamoDB database service's regional endpoint, which caused that endpoint's DNS record to be emptied out
Requs AI Edge Case flags this exact pattern at the requirements and architecture stage — before a single line of code implementing it exists — so the assumption behind it gets challenged while it is still cheap to fix.
Every service that needed a fresh connection to DynamoDB...
every service that needed a fresh connection to DynamoDB in that region began failing DNS resolution
Requs AI Software FMEA traces this failure mode back to the system-level hazard it feeds, tagging it against the Common Defect Enumeration so it surfaces in review instead of in the field.
The failure then cascaded into EC2's instance-launch subsystem and...
the failure then cascaded into EC2's instance-launch subsystem and Network Load Balancer health checks, prolonging the recovery
Requs AI Edge Case and Software FMEA together treat this as a known, catalogued defect pattern — not a novel surprise — so it gets tested for deliberately rather than discovered after deployment.
Beyond code coverage and "shall" testing
Root causes like this one rarely show up in code coverage or requirements-compliance testing, because nobody wrote a requirement anticipating the specific edge case that broke. Requs AI Edge Case and Requs AI Software FMEA are built to surface exactly this class of overlooked failure mode — before the software is written.
Surfaces this before code exists
Identifies edge cases like this one at requirements and architecture time, using the Common Defect Enumeration to catalog failure patterns seen across hundreds of real-world software failures — including this one.
Connects the failure mode to the hazard
Traces this class of root cause directly to the system-level hazard it can produce, so a defect pattern like this one gets flagged during design review instead of after it ships.
Find the overlooked root causes before they ship.
Schedule a demonstration, or explore how Requs AI Edge Case and Software FMEA use the Common Defect Enumeration.