The Kill Switch Problem

We build safety mechanisms and then quietly disable them. The alert that pages too often gets a filter. The automation that touched something important once gets commented out.

That's the problem with a kill switch. There will be a morning when cost-guard stops something I needed, and I'll reach for one of two things: the command that puts it back, or the switch that kills the tool forever. Which one I reach for gets decided by what I allowed the tool to do, long before that morning.

So the constraint came first. Nothing irreversible. Nothing touched without a tag somebody put there deliberately. Dry run by default.

Turning things off was never the hard part. Six lines set a service's desired count to zero. The week went into everything that happens after.

What AWS Already Does

Budgets has actions built in, and it's worth an hour finding out what they are. Most side projects that die of pointlessness died because nobody spent it.

Budgets can attach an IAM policy, attach an SCP, or stop EC2 and RDS instances, and any of that can be reversed. But the stop action wants instance IDs listed up front, and the policy actions block new spend while leaving everything already running exactly where it is.

Nothing in there pauses a workload it wasn't told about in advance. That's the gap. It's narrow, and narrow is fine.

The Shape

budget alert -> SNS -> Lambda -> tagged resources
                        |
                        +-> DynamoDB (what they were)
                        +-> SNS (report)

Two topics, not one. The function publishes a report on every run, and if that report went to the topic that triggers the function, I'd have built a machine that wakes itself up forever. Two topics makes the loop impossible rather than merely unlikely.

Budgets needs permission to publish, which lives in the topic policy:

data "aws_iam_policy_document" "trigger_topic" {
  statement {
    actions   = ["sns:Publish"]
    resources = [aws_sns_topic.trigger.arn]

    principals {
      type        = "Service"
      identifiers = ["budgets.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
}

The budget resource itself is unremarkable. Its depends_on isn't. Without it Terraform is free to create the budget before that policy exists, and the subscription fails. The same trap waits on the Lambda side, where the subscription can be created before the function will accept anything from the topic:

resource "aws_sns_topic_subscription" "lambda" {
  topic_arn = aws_sns_topic.trigger.arn
  protocol  = "lambda"
  endpoint  = aws_lambda_function.this.arn

  depends_on = [aws_lambda_permission.trigger]
}

Terraform builds what you describe. It doesn't infer what you meant.

The Limits of a Snapshot

Every resource type is a module with three functions: find, pause, resume. The dispatcher is a dictionary. That part wrote itself.

Here's the part that didn't. Before pausing anything, write down what it was running. Read that back to restore. Simple enough to feel finished.

Then run it twice.

Somebody scales a service back up by hand on Tuesday. The budget trips again on Wednesday. The function looks at that service and dutifully records what it sees. But what it sees on Wednesday isn't what the service was. It's whatever a person set it to in the middle of an incident. The original number is gone, quietly, and you find out on the day you restore and get back a configuration nobody ever chose.

The fix is a conditional write:

def remember(table, kind, resource_id, snapshot):
    try:
        table.put_item(
            Item={
                "resource_id": resource_id,
                "kind": kind,
                "snapshot": json.dumps(snapshot),
                "paused_at": datetime.now(timezone.utc).isoformat(),
            },
            ConditionExpression="attribute_not_exists(resource_id)",
        )
    except ClientError as err:
        if err.response["Error"]["Code"] != "ConditionalCheckFailedException":
            raise
        # Someone scaled it back up after an earlier pause. The first snapshot
        # holds the real original settings, so keep it.
        return False
    return True

The bug wasn't in the write, and it wasn't in the restore. Both were correct. It lived in the space between them, in the assumption that the world holds still while you work.

The Order of Operations

Pause first, then record: a crash in between leaves something stopped and no note saying what it was.

Record first, then pause: a crash leaves a note for something that was never paused, and a later restore acts on a lie.

Both are bad. They aren't equally bad. The first can't be recovered and the second is a wrong number I can find and delete, so the order is settled on purpose rather than by the accident of which line I typed first.

try:
    created = remember(table, kind, resource_id, snapshot)
except ClientError as err:
    lines.append(f"FAILED to record {kind} {resource_id}, leaving it alone: {err}")
    continue

try:
    module.pause(resource_id, snapshot)
except ClientError as err:
    if created:
        table.delete_item(Key={"resource_id": resource_id})
    lines.append(f"FAILED to pause {kind} {resource_id}: {err}")
    continue

Look at the first block. If DynamoDB won't take the note, the resource keeps running. Overspending is a bill. Pausing something I can't restore is a hole.

What the Services Do Behind You

Auto Scaling groups relaunch what you stop. Set desired capacity to zero and the group builds replacements to reach its minimum. You haven't paused it, you've given it something to do. So min comes down with desired, and the snapshot has to carry min, max, and desired rather than desired alone.

RDS restarts itself. AWS allows a stopped instance for seven days, then starts it for you, whether or not you were done saving money. Restore can't simply call start_db_instance, because it'd fail against a running database and leave the row in the table forever:

def resume(db_arn, snapshot):
    rds = boto3.client("rds")
    status = rds.describe_db_instances(DBInstanceIdentifier=snapshot["identifier"])["DBInstances"][0][
        "DBInstanceStatus"
    ]
    # AWS starts a stopped instance by itself after seven days
    if status in ("available", "starting"):
        return
    rds.start_db_instance(DBInstanceIdentifier=snapshot["identifier"])

Lambda has a null case worth respecting. Reserved concurrency of zero and reserved concurrency never set are different states wearing similar clothes, so restoring a function that never had one means deleting the setting rather than writing a number. The handler also refuses to touch its own function, since a provider-wide default_tags block would otherwise hand the tool a gun and point it at the tool.

None of these are edge cases. They're the system having intentions of its own, which it always does.

Testing Against moto

moto fakes the AWS APIs in process, so the tests need no credentials and run in CI. That's convenient. The real reason to use it is that the failure paths become testable, and the failure paths are the entire point. Producing an AccessDenied against real AWS means breaking your own permissions on purpose and remembering to put them back.

def test_second_pause_keeps_the_original_count(service):
    service("web", desired=3)
    app.handler({}, None)
    boto3.client("ecs").update_service(cluster="apps", service="web", desiredCount=1)

    app.handler({}, None)
    assert desired_count("web") == 0

    app.handler({"action": "restore"}, None)
    assert desired_count("web") == 3

That's Tuesday and Wednesday, written down. I confirmed the RDS test the same way, by reverting the fix and watching it go red. A test you've never seen fail is a decoration.

What none of it tells me is whether the IAM policy is sufficient. moto doesn't enforce permissions, so that answer lives in a real account and nowhere else.

What Can't Be Fixed

AWS refreshes budget data a few times a day, 8 to 12 hours apart. Something can burn half a day of money before the function hears a word about it. No cleverness on my side changes that, so the README says the plain thing: a backstop, not a circuit breaker.

That sentence is worth more than it looks. A tool that tells you what it can't do is a tool you can reason about. A tool that stays quiet about its limits gets trusted with something it was never able to do, and the silence costs more than the admission would have.

The rest of the list is short. One region. Restore is all or nothing. A scaling policy can undo a pause five minutes after it happens.

Where That Leaves Us

Almost every answer in that week turned out to be a single line. The week went into finding the questions, and there was really only one, asked over and over: what happens if the step after this one never runs?

A tool you trust isn't one that never fails. It's one whose failures you already know the shape of.

Code is at terraform-aws-cost-guard, MIT, and deliberately narrow. It's for accounts with workloads you'd rather have off than spendy.