This is the Principal-level systems thinking question. Structure, speed, and communication matter as much as technical depth.

Incident response methodology

Phase 1 — Triage (first 2 minutes):

uptime; free -h; df -h; ss -s # four commands, system-wide health

dmesg | tail -20 # any kernel errors?

systemctl list-units --failed # any failed services?

Phase 2 — Identify bottleneck (next 5 minutes):

vmstat 1 5 # r/b columns: CPU queue or I/O blocked?

top -b -n 1 # which process is the culprit?

iostat -xz 1 3 # which disk is saturated?

iftop # which connection is saturating network?

Phase 3 — Confirm hypothesis:

strace -c -p <pid> sleep 5 # what syscalls is the process making?

lsof -p <pid> # what files/sockets are open?

journalctl -u <service> --since "10 min ago" # service logs

Phase 4 — Immediate relief + root cause fix:

# Immediate: kill/restart/scale the problematic component

# Root cause: fix config, add resource, fix code

Phase 5 — Validate + document:

# Confirm metrics return to baseline

# Write postmortem: timeline, root cause, fix, prevention

Common patterns by symptom: high load + high I/O wait + D-state processes = storage bottleneck (check iostat await). High load + low I/O wait + high %sy = iptables/networking overhead (many rules, high connection rate). High load + low I/O wait + high %us = CPU-bound application (profile with perf). Memory exhausted + swap in use = OOM risk imminent (free up memory or add swap). Filesystem full + lsof shows deleted files = truncate open files. SSH not working but ping works = service issue, firewall, or PAM problem.

Principal-level deliverable: After every incident: write a postmortem (5 whys root cause analysis), add the missing alert that would have caught this earlier, and improve the runbook. Convert reactive incidents into proactive prevention. The best Platform Engineers make their own on-call rotation quieter over time — not because they silence alerts, but because they fix the underlying issues permanently.