HOW A MISCONFIGURED LOAD BALANCER TOOK DOWN OUR APPLICATION

The Setup

Our application runs across three availability zones behind an AWS Application Load Balancer (ALB). At the time of the incident, we were handling approximately 3,400 requests per second at peak, distributed across 18 backend EC2 instances in an auto-scaling group.

The platform team had been in the middle of a routine infrastructure hardening initiative — tightening health check parameters, reducing idle connection timeouts, and enforcing stricter SSL policies across all load balancers. Tickets had been open for weeks. It was a Tuesday afternoon maintenance window. Everything seemed low-risk.

It was not low-risk.

What Actually Happened

At 14:32 UTC, an engineer applied a Terraform plan to update the ALB target group health check configuration. The change was intended to reduce the health check interval from 30 seconds to 10 seconds — a sensible tightening to speed up unhealthy instance detection.

But bundled in the same Terraform apply, almost unnoticed, was a second parameter change: the health check timeout was reduced from 5 seconds to 2 seconds.

Our Node.js /health endpoint — under normal load — typically responds in 80–140ms. But under the load our application was experiencing at 14:32 UTC, with a cold database connection pool warming up after a scheduled maintenance restart 20 minutes earlier, the health endpoint was occasionally taking 2.1 to 2.8 seconds to respond.

A timeout of 2 seconds meant those responses were being seen as failures by the ALB — even though the instances were perfectly healthy.

The Error Logs at 14:34 UTC

The Cascade — Why It Spread So Fast

The insidious part was the feedback loop. As instances were marked unhealthy and removed from the target group, the remaining healthy instances absorbed more traffic. Under increased load, their /health endpoints also started responding slowly — breaching the 2-second timeout threshold. They too were marked unhealthy.

Auto Scaling tried to help. It launched new instances. But new instances, cold-starting with a fresh database connection pool, had worse health check latency than the running instances — and were immediately marked unhealthy before they could even serve traffic.

Within 2 minutes and 37 seconds of the first timeout, all 18 instances were marked unhealthy. The ALB had zero healthy targets and returned 503 to every request.

Minute-by-Minute Incident Timeline

• 14:32:01

  Terraform apply executed

  ALB health check interval changed to 10s, timeout changed to 2s. Engineer did not flag timeout change in PR as it appeared to be a minor tightening. Change goes live immediately.

• 14:32:45

  First health check timeout detected

  One instance's /health responds in 2.14s. ALB records first failure. No alert fires yet — 3 consecutive failures required to mark unhealthy.

• 14:33:10

  First instances marked unhealthy — cascade begins

  Two instances hit 3 consecutive failures. Removed from rotation. Remaining instances see 12% more traffic each. Their health check latency begins climbing.

• 14:34:00

  Auto Scaling launches new instances — makes it worse

  New instances start with cold DB connection pools. Health check latency on new instances: avg 2.9s. They are marked unhealthy before serving a single request.

• 14:34:38

  Zero healthy targets — full outage begins

  All 18 instances marked unhealthy. ALB returns 503 to 100% of requests. 220,000 active users see error pages. PagerDuty fires.

• 14:35:01

  On-call engineer engaged

  Initial investigation focuses on application errors, not infra. 4 minutes lost checking deployment logs and application metrics before load balancer is examined.

• 14:39:22

  Root cause identified

  Engineer correlates ALB health check failures with Terraform change at 14:32. The timeout parameter change is spotted in the Terraform plan diff. Remediation plan formed.

• 14:42:00

  Emergency timeout increase applied via AWS Console

  Health check timeout changed from 2s to 10s directly in AWS console (bypassing Terraform for speed). Health checks begin passing. First instances recover.

• 14:52:00

  Full recovery — all instances healthy

  All 18 instances pass health checks. Error rate returns to baseline. Total outage duration: 47 minutes 22 seconds. Post-incident monitoring engaged.

The Exact Configuration That Broke Us

Here is the precise Terraform diff that caused the incident. The interval change was the intended change. The timeout change was the silent killer — it looked like a routine tightening but created an impossible constraint for our application under load.

The broken timeout of 2 seconds was actually lower than the p99 response time of the health endpoint under peak load. This meant that during the highest-traffic periods — exactly when reliability mattered most — health checks were structurally guaranteed to fail.

The Correct Health Check Endpoint (Post-Incident)

// AFTER: Lightweight health check — no DB dependency
// Responds in <5ms under any load condition

app.get('/health', (req, res) => {
  // Check only in-process state — never external dependencies
  const status = {
    status:   'ok',
    uptime:   process.uptime(),
    memory:   process.memoryUsage().heapUsed,
    version:  process.env.APP_VERSION
  };

  res.status(200).json(status);
});

// Separate readiness endpoint for deep checks (not used by ALB)
app.get('/ready', async (req, res) => {
  try {
    await db.query('SELECT 1');
    await redis.ping();
    res.status(200).json({ ready: true });
  } catch (err) {
    res.status(503).json({ ready: false, error: err.message });
  }
});

Six Things We Got Wrong

What We Changed After the Incident

• POLICY
  OPA policy rule: health check timeout floor Added an Open Policy Agent rule that rejects any Terraform plan setting an ALB health check timeout below 10 seconds. Applied across all environments in CI/CD.
• CODE
  Lightweight /health endpoint — no external calls Refactored all service health check endpoints to return a static JSON response in under 5ms. Deep dependency checks moved to a separate /ready endpoint not used by ALB.
• INFRA
  ALB timeout set to 10× P99 health check latency Established a rule: health check timeout = max(10s, 10× measured P99 health endpoint response time). Re-evaluated every quarter and after major application changes.
• ALERT
  Direct alert on ALB healthy host count Added CloudWatch alarm on HealthyHostCount that fires as soon as any instance is marked unhealthy. Runbook linked directly in the alert for immediate diagnosis.
• SCALE
  Instance warm-up period in Auto Scaling Configured a 90-second warm-up period before new Auto Scaling instances are counted in health check evaluations. Prevents cold-start kills during scaling events.
• REVIEW
  Required review label for health check changes Any Terraform change touching health_check blocks now auto-assigns a required reviewer from the Platform team. Changes cannot merge without explicit sign-off.

The One-Line Takeaway

A 47-minute outage affecting 220,000 users came down to a single number: 2 seconds instead of 5. Not a complex distributed systems failure. Not a novel attack. A field in a config file that nobody thought to stress-test under realistic conditions.

The lesson isn't to be more careful with config changes — it's to build systems where a single config change cannot cause a full outage. Guardrails, policy enforcement, staged rollouts, blameless culture, and deep observability are the layers that transform a catastrophic incident into a minor blip.

We've made all of those changes. The next misconfiguration will hit a guardrail long before it hits your users.