Deploying a Lambda function to periodically delete EC2 instances

I'm Teraoka, an infrastructure engineer.
This time, I'd like to summarize my experience deploying a Lambda function for internal use.

In addition to the AWS accounts where our clients' servers run under our operational management,
we also manage accounts used by our in-house engineers for technical verification.

This account has defined rules for login security settings (IAM user roles/MFA authentication), but
operations within the account are generally unrestricted for members with the necessary permissions.
Too many rules would make it difficult to use, especially for testing purposes.

In our case, the most frequently created resource during the verification process is EC2 instances.
While they can be created freely, this also meant that forgetting to stop or delete them was a fairly common occurrence.

Naturally, frequent occurrences result in unnecessary costs.
While the premise is that instances should always be stopped or deleted after use,
we have implemented a mechanism to automatically delete EC2 instances periodically using Lambda as a precaution in case this is forgotten.

So below is a summary of what I did

Creating a Lambda function

I wrote it in Python.
Personally, I'm more familiar with Golang, so I feel like I should have written it in that instead.

import json import boto3 import os import requests def terminate(event, context): EXCLUDE_TERMINATE_TAG = os.environ['EXCLUDE_TERMINATE_TAG'] CHATWORK_API_KEY = os.environ['CHATWORK_API_KEY'] CHATWORK_ROOM_ID = os.environ['CHATWORK_ROOM_ID'] CHATWORK_ENDPOINT = os.environ['CHATWORK_ENDPOINT'] ec2 = boto3.client('ec2') without_tag = ec2.describe_instances() with_tag = ec2.describe_instances( Filters=[{ 'Name': 'tag:' + EXCLUDE_TERMINATE_TAG, 'Values': ['true'] }] ) without_tag_set = set(ec2['InstanceId'] for resId in without_tag['Reservations'] for ec2 in resId['Instances']) with_tag_set = set(ec2['InstanceId'] for resId in with_tag['Reservations'] for ec2 in resId['Instances']) target_instances = without_tag_set - with_tag_set list_target_instances = list(target_instances) if len(list_target_instances) != 0: terminateInstances = ec2.terminate_instances( InstanceIds=list_target_instances ) notify_instances = '' if len(with_tag['Reservations']) != 0: for reservation in with_tag['Reservations']: for instance in reservation['Instances']: if len(instance['Tags']) != 0: instance_name = '' for tag in instance['Tags']: if tag['Key'] == 'Name': instance_name = tag['Value'] instance_state_enc = json.dumps(instance['State']) instance_state_dec = json.loads(instance_state_enc) if instance_name != '': notify_instances += instance['InstanceId'] + ' -> ' + instance_name + '(' + instance_state_dec['Name'] + ')'+ '\n' else: notify_instances += instance['InstanceId'] + ' -> ' + 'None' + '(' + instance_state_dec['Name'] + ')'+ '\n' message = '[To:350118][To:1786285][info][title][Beyond POC] There are retired instances (devil)[/title]' + notify_instances + '[/info]' PostChatwork(CHATWORK_ENDPOINT,CHATWORK_API_KEY,CHATWORK_ROOM_ID,message) response = { "TerminateInstances": list_target_instances } print (response) return response def PostChatwork(ENDPOINT,API_KEY,ROOM_ID,MESSAGE): post_message_url = '{}/rooms/{}/messages'.format(ENDPOINT, ROOM_ID) headers = { 'X-ChatWorkToken': API_KEY } params = { 'body': MESSAGE } resp = requests.post(post_message_url, headers=headers, params=params)

I think you can understand if you look at the code

• Delete instances that do not have an exclusion tag
• Notify Chatwork of instances that have been excluded from deletion

This process is performed for us.
However, since it can be problematic if files are deleted without question,
in such cases, you should add the exclusion tag "EXCLUDE_TERMINATE: true"
we have a rule within the company that

Serverless Framework configuration

This talk will explain how to deploy the code you write to Lambda

We are using a tool called Serverless Framework.
https://serverless.com

to serverless services such as Lambda
a CLI tool that deploys functions
, and in the case of Lambda, it works in conjunction with CloudFormation to perform the deployment.

Prepare a configuration file named serverless.yml

# Welcome to Serverless! # # This file is the main config file for your service. # It's very minimal at this point and uses default values. # You can always add more config options for more control. # We've included some commented out config examples here. # Just uncomment any of them to get that config option. # # For full config options, check the docs: # docs.serverless.com # # Happy Coding! service: beyond-poc plugins: - serverless-prune-plugin provider: name: aws runtime: python3.8 profile: ${opt:profile, ''} region: ap-northeast-1 role: [IAM Role Name] environment: EXCLUDE_TERMINATE_TAG: EXCLUDE_TERMINATE CHATWORK_API_KEY: [ChatWark API KEY] CHATWORK_ROOM_ID: [ChatWark ROOM ID] CHATWORK_ENDPOINT: https://api.chatwork.com/v2 timeout: 10 # my custom env custom: prune: automatic: true number: 5 # you can overwrite defaults here # stage: dev # region: us-east-1 # you can add packaging information here #package: # include: # - include-me.py # - include-me-dir/** # exclude: # - exclude-me.py # - exclude-me-dir/** functions: terminate-instance: handler: handler.terminate events: - schedule: cron(0 15 ? * * *) timeout: 120 memorySize: 128 reservedConcurrency: 1

In the `environment` section, I've set exclusion tags and ChatWork authentication information as environment variables.
In the `functions` section, I've literally set up functions.
In the `events` section, I've set up a schedule, which,
with this setting, will run periodically every day at 15:00 (UTC) in conjunction with CloudWatch Events.
In JST, that would be midnight (0:00).

Save the code to be deployed in the directory where this configuration file is located

$ sis deploy --profile [AWS Profile Name]

automatically created by the Serverless Framework from the command line
you can execute and deploy the CloudFormation stack

lastly

I'm shamelessly using it as blog material, but
actually, I'm the one who forgets to delete things the most in the company. Sorry about that...