Episode summary
An early-stage startup spins up an S3 bucket for user-uploaded images. They are moving fast, the policy is permissive, the bucket is public-read. A misconfigured client app starts hammering the bucket from a third-party CDN's IP range that does not get cached.
The bill that arrives next month is $84,000.
This episode is about what happened, what the founder did when AWS would not forgive the bill, and the three controls that would have caught it before it ran for 27 days. We walk through the actual CUR data (anonymized) and the CloudFront configuration the company should have had from day one.
The story has a happy ending, kind of. AWS forgave 60 percent of the bill. The remaining $33K was paid out of a Series A bridge round. The startup is still alive. The founder no longer touches S3 settings without a checklist.
Show notes
- Why request-payer is the most under-used S3 feature.
- A four-line CloudWatch alarm that would have caught this on day three.
- The sequence of conversations with AWS billing support, transcribed.
- Lifecycle policies you should have on every bucket you spin up.