The 504 That Wasn't My Server: A CloudFront VPC Origin Outage

One morning the site was down. Every page returned a 504 Gateway Timeout, and my first instinct was the one everyone reaches for: the little t4g.small must have run out of memory again.

It hadn't. And chasing that assumption would have burned the whole morning.

The server was innocent

First checks on the instance: load 0.02, plenty of free RAM, disk at 54%, no out-of-memory kills in the kernel log, Apache and PHP-FPM both up, 64 days of uptime. Then the check that actually mattered — I curled the app locally on the box:

HTTP 200 in 0.09s

The application was healthy and fast. The 504 was coming from somewhere between CloudFront and the server — not from the server itself. That single local curl reframed the entire investigation.

The architecture, briefly

The site sits behind CloudFront. Instead of a public origin or a load balancer, it used a CloudFront VPC Origin — a newer feature where CloudFront reaches a private EC2 instance through an AWS-managed network card (an ENI) placed inside your VPC. No public IP, no ALB. Clean, and locked down.

That ENI is the one piece you don't control and can't log into. Which made it the prime suspect.

Proving it with tcpdump

Suspicion isn't evidence. So I ran tcpdump on the instance, filtered to the VPC-origin ENI's IP, and fired live requests at CloudFront while it captured. The result was decisive: zero packets reached the instance while CloudFront was timing out against it.

Then I ruled out the boring culprits, one by one:

• Opened the instance's security group to the entire VPC — still zero packets.
• ENI and instance in the same subnet, with a default allow-all network ACL, and no host firewall.
• Hit the instance directly over its public IP — clean 200.

So the server accepts traffic fine, but packets from CloudFront's VPC-origin ENI simply never arrive. The break was inside AWS's managed forwarding path — nothing I had configured, and nothing I could fix from my side. To be sure, I recreated the VPC origin from scratch. Identical failure.

The fix: reach for AWS's own escape hatch

With the fault isolated to the VPC-origin path, the fastest safe fix was to stop using it. I switched the CloudFront distribution to a public custom origin (the instance's public endpoint) and locked the instance's firewall to CloudFront's managed origin-facing IP prefix list — so the origin is reachable only from CloudFront's edge servers, never the open internet. Site back to 200 within minutes.

An hour later, the reply on my support case said to do exactly that: "customers who are able to do so [should] temporarily change their origin type." I'd already shipped their recommended workaround before they sent it.

AWS confirmed it

The Service Health Dashboard eventually posted the incident: increased 5xx errors for CloudFront customers using VPC Origins, in our region, for a few hours that morning. Their stated root cause was an internal fleet constraint that caused the system distributing routing configuration to their network processors to fail to load — breaking edge-to-VPC routing. Customers on other origin types were unaffected.

That matched my capture precisely: the packets never arrived because the thing responsible for routing them had stopped working.

Once AWS marked it resolved, I reverted — back to the private VPC origin, public exposure removed — as a zero-downtime, two-step swap: bring the VPC path back and confirm 200 while both paths were open, then close the public one.

What I took away

• A 504 is a conversation between two machines. Before blaming the server, prove which side has gone quiet. A curl on localhost and a tcpdump for the expected source told me more than any amount of guessing.
• Isolate before you fix. The tempting move was to reboot Apache and hope. The useful move was to confirm the server was serving and that the packets weren't arriving.
• Managed features still fail — keep an escape hatch. VPC Origins are elegant, but when the managed layer broke, recovery meant falling back to the plainer, battle-tested public-origin path. Knowing that fallback existed turned an outage into a 15-minute change.
• Preserve the evidence. I left the broken VPC origin in place, untouched, so AWS could investigate — and only reverted after they confirmed the fix.

Working through it with an AI pair helped most on the discipline, not the trivia: it kept me testing the "is it even reaching the box?" hypothesis instead of settling for the first plausible story. The server wasn't out of memory. On this one, it barely had anything to do with the outage at all.