Failover that's never tested usually fails when it matters. Designing it means accounting for detection time, convergence, and state — then proving it with drills.

The components of failover time

Detection : how fast you notice failure (health-check interval x threshold) Convergence: how fast traffic reroutes (DNS TTL, BGP, gratuitous ARP, LB dereg) State : what's lost on cutover (in-flight requests, sessions, sticky state) RTO/RPO : the actual downtime + data-loss you can tolerate

Where failover silently doesn't work

- DNS failover blocked by long client/JVM caching (TTL ignored) -> slow/no cutover - Stale ARP/conntrack after VIP move -> peers keep hitting the dead node - Standby was never actually in sync / never tested -> promotes to bad state - Health check too lax (detects failure minutes late) or too aggressive (false failover)

Verify it

- GAME DAYS: actually kill the primary / a zone and measure real RTO/RPO - Chaos testing: inject failures continuously, not once a year - Monitor failover events; alert if the standby is unhealthy/out of sync

Gotcha: The most common failure is assuming a configured failover works — DNS TTL caching, unsynced standbys, and stale ARP/conntrack routinely defeat it. Detection + convergence often add up to far more than the "60s TTL" you assumed. Untested failover is a liability, not a safety net.

Interview tip: Break failover into detection + convergence + state, and name the silent killers (DNS/JVM caching defeating TTL, stale ARP/conntrack, unsynced standby). The headline: test it with game days / chaos engineering and measure real RTO/RPO — "we have failover configured" is meaningless until you've pulled the plug and watched it recover. This maturity is exactly what senior/SRE interviews probe.