TG
All posts
2 min read

Engineering Self-Healing AWS Pipelines: Slashing MTTR by 65%

An SNS fan-out that sends degradation to an idempotent Lambda runbook for repair and to Slack for the record — with the guardrails that stop it destroying healthy capacity.

A pager at 3 AM, a login, a look at the logs, a manual restart: every one of those steps is time the service is still down. Mean Time To Recovery is the number that measures it, and human intervention sets a floor under it that no amount of dashboard-watching removes.

So we stopped making a human the remediation path. Alerts still reach Slack — you always want the record — but they are no longer what fixes anything. A failing health check invokes an automated runbook directly, and the Slack message is a notification of work already done.

EVENT-DRIVEN REMEDIATION ON AWSDEGRADATION DETECTEDEVENT ROUTINGAUTOMATED RUNBOOKCORRECTIVE ACTIONAUDIT TRAILEC2 / ASG nodehealth check or memoryElastic Load Balancermarks node unhealthyCloudWatch alarmstate changes to ALARMSNS topicfans out the payloadAWS LambdaBoto3, strictly idempotentIAM roleleast privilegeEBS snapshottaken before replacementReplace instancetag AutoRemediate=TrueAmazon RDSpoint-in-time recoverySlack webhookinstance, alarm, action taken1. fails health check2. marked unhealthy3. alarm published4. invoke runbookscoped by tag5. snapshot volume6. replace nodestate untouchednotify, do not remediatenumbered happy pathguardrail / audit path
SNS fans out deliberately: Lambda does the remediation, Slack only records it. The alert is not the response.

How a failure resolves itself

  1. An EC2 node in the Auto Scaling group fails its Elastic Load Balancer health check, or exhausts memory.
  2. The load balancer marks it unhealthy and stops routing to it.
  3. The corresponding CloudWatch alarm changes state to ALARM and publishes to an SNS topic.
  4. SNS fans the payload out. One branch invokes the Lambda runbook, written in Python against Boto3.
  5. The runbook takes an EBS snapshot of the instance before touching it.
  6. Only then does it replace the node.

The second SNS branch posts to a Slack webhook with the instance, the alarm and the action taken. That branch notifies; it does not remediate. Keeping the two separate is what stops an outage waiting on someone reading a channel.

The guardrails that make automation safe

Idempotency. Before acting, the runbook calls the EC2 API and reads the instance's current state — pending, running, or stopping. If several alarms fire together during a network blip, a runbook without that check races itself and terminates the healthy replacement node it just created.

Blast radius. The function's IAM role is scoped to instances carrying the AutoRemediate: True tag. An automation bug cannot reach anything that has not explicitly opted in.

Data resilience. Replacing compute is only safe if the data survives it. The database tier runs on Amazon RDS with Point-in-Time Recovery continuously enabled, so the remediation path never touches state — and the EBS snapshot in step 5 means even the replaced instance's disk is recoverable.

We validated all of this by running live recovery drills against the parameterized CloudFormation stack rather than trusting it in theory. Those drills confirmed a sub-5-minute Recovery Point Objective and a 65% reduction in MTTR.

GitHub
LinkedIn