Automatically generate thumbnail images triggered by S3 image upload using AWS lambda
My name is Teraoka and I am an infrastructure engineer.
The theme this time is "lambda", one of the AWS services.
AWS Lambda (serverless code execution and automatic management) | AWS
By the way, it is pronounced "lambda". Where did the "b" in between go?
That being said,
there are already a lot of articles like this, and I don't know how many will be covered, but
I decided to try out a little bit for my own study.
This time, we will use lambda to create a thumbnail image. Please see below for details ↓
■First of all, what is “lambda”?
Yes, first of all, let's start with an overview of lambda.
As usual, I will quote from the official AWS documentation (
What is AWS Lambda? - AWS Lambda - AWS Documentation
AWS Lambda is a computing service that lets you run code without provisioning or managing servers.
...I see.
can execute pre-registered code
using an event as a trigger Additionally, since this event is used as a trigger to execute processing asynchronously,
there is no need to constantly start up EC2 instances, etc.
This is the part where you don't have to provision or manage servers.
Now, the word "event" has been mentioned earlier, but
in a nutshell, it is as follows.
- File uploaded to S3 bucket
it is possible to
automatically execute some process when a file is uploaded to an S3 bucket Of course, I won't go and attack the API on the EC2 instance.
It is completed with just S3 and lambda.
Yes, if you have read the content and blog title up to this point, you probably got the hang of it, but
this time I will introduce ``Automatically generate a thumbnail image of an image when a file is uploaded to an S3 bucket''
... I would like to do this (
■The path to creating thumbnail images
- Prepare an S3 bucket to upload images
- Writing a lambda function
- Function test
- Configuring lambda trigger
- Operation confirmation
① Prepare an S3 bucket to upload images
First, we can't start without preparing a container, so we'll quickly create a bucket in S3.
Please see the article below for creating a bucket and setting policies.
This is something I wrote before, so I'll blurt it out and promote it (
②Description of lambda function
Now it's time to write the lambda function.
Write the code to "automatically generate thumbnail images" and register it with lambda.
Regarding the code, it can be written based on python or node.js, but
this time I will respect my personal preference and write it in node.js (sorry to those who like python.
First, click the button to create a Lambda function. Click on it, and you will see a screen like the one below.
Select the lambda function blueprint on this screen.
I would be grateful if you could use the templates that have been prepared in advance to suit your needs.
Select "Node.js6.10" from the runtime selection.
As for the blueprint, I want to perform processing related to s3, so I will use something called "s3-get-object".
Click on the blueprint...
The trigger setting screen will look like this.
bucket | Select the bucket you created in ①. |
---|---|
event type | This time, we want to execute the function "Once the image is uploaded", so select Put. |
prefix | This time, we will upload directly under the bucket, so we will omit it. |
suffix | Only files whose file names end with "jpg" will be executed by the function. |
Do not check enable trigger.
Enable it manually after checking the operation of the function that will be written later.
When you have finished entering your information, click Next.
We will actually write the function on this screen.
There is a field to enter the function name, but you can use any name you like.
Since we selected "s3-get-object" on the blueprint selection screen,
the code to retrieve the file from S3 has been written in advance.
'use strict'; console.log('Loading function'); const aws = require('aws-sdk'); const s3 = new aws.S3({ apiVersion: '2006-03-01' }); exports. handler = (event, context, callback) => { //console.log('Received event:', JSON.stringify(event, null, 2)); // Get the object from the event and show its content type const bucket = event.Records[0].s3.bucket.name; const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' ')); const params = { Bucket: bucket, Key: key, }; s3.getObject(params, (err, data) => { if (err) { console.log(err); const message = `Error getting object ${key} from bucket $ {bucket}. Make sure they exist and your bucket is in the same region as this function.`; console.log(message); callback(message); } else { console.log('CONTENT TYPE:', data.ContentType ); callback(null, data.ContentType); } }); };
However, if this continues, you will be able to retrieve files from S3, but you
will not be able to generate thumbnail images.
I'll add a little bit to this code.
'use strict'; console.log('Loading function'); var fs = require('fs'); var im = require('imagemagick'); const aws = require('aws-sdk'); const s3 = new aws.S3({ apiVersion: '2006-03-01' }); exports.handler = (event, context, callback) => { const bucket = event.Records[0].s3.bucket.name; const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' ')); const params = { Bucket: bucket, Key: key, }; s3.getObject(params, ( err, data) => { if (err) { console.log(err); const message = `Error getting object ${key} from bucket ${bucket}. Make sure they exist and your bucket is in the same region as this function.`; console.log(message); callback(message); } else { var contentType = data.ContentType; var extension = contentType.split('/').pop(); console.log(extension); im.resize({ srcData: data.Body, format: extension, width: 100 }, function(err, stdout, stderr) { if (err) { context.done('resize failed', err); } else { var thumbnailKey = key.split('.')[0] + "-thumbnail." + extension; s3.putObject({ Bucket: bucket, Key: thumbnailKey, Body: new Buffer(stdout, 'binary'), ContentType: contentType }, function(err, res) { if (err) { context.done('error putting object', err); } else { callback(null, "success putting object"); } }); } }); } }); };
Once added, it will look like this.
The process is the same until the file is retrieved from S3, but
the retrieved image is resized using imagemagick and the
processed image is uploaded to S3 again as a thumbnail image.
After writing the code, configure the IAM Role as shown below.For
now, let's skip this part and automatically create a new Role from the template.
The policy for manipulating S3 objects will be attached to the role created here, so
if you do not set it correctly, the function may result in an error.
(Access Denied is a common problem due to lack of access privileges to S3)
When you click Next, a confirmation screen will appear, so press the "Create function" button.
Then a lambda function will be created.
The function you just created is displayed in the list!
③Function operation test
Test whether the function you created works correctly.
Select Test Event Settings from the actions at the top of the screen.
The test event input screen will then appear.
The test event is written in json format, but
please consider that lambda receives this json from S3 and executes the function.
we will test whether the lambda function works properly by assuming that we have uploaded an image file called test.jpg to an S3 bucket in advance and
A template is also provided for this test event, so
let's use the template called S3 Put.
Enter the following and click Save and Test.
{ "Records": [ { "eventVersion": "2.0", "eventTime": "1970-01-01T00:00:00.000Z", "requestParameters": { "sourceIPAddress": "*" }, "s3": { "configurationId": "testConfigRule", "object": { "eTag": "0123456789abcdef0123456789abcdef", "sequencer": "0A1B2C3D4E5F678901", "key": "test.jpg", "size": 1024 }, "bucket" : { "arn": "arn:aws:s3:::lambda-img-resize", "name": "lambda-img-resize", "ownerIdentity": { "principalId": "EXAMPLE" } }, " s3SchemaVersion": "1.0" }, "responseElements": { "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH", "x-amz-request-id": "EXAMPLE123456789" }, "awsRegion": "ap-northeast-1", "eventName": "ObjectCreated:Put", "userIdentity": { "principalId": "EXAMPLE" }, "eventSource": "aws:s3" } ] }
A message was returned when the execution was successful!
Let's check if there are any changes in the contents of the S3 bucket.
Yes, the thumbnail image is being generated as if nothing had happened.
There seems to be no problem with the function execution itself.
④Lambda trigger settings
Finally, in the final stage, we will set the trigger.
The previous step was just a test, so
even if you upload a new image at this stage, a thumbnail image will not be created.
You need to set up a trigger so that the function runs automatically when the image is uploaded.
Click the Trigger tab from the lambda function you created.
The trigger itself was created when creating the function, so all that is left is to enable it.
Click Activate.
⑤Operation confirmation
Now, let's actually upload the image to S3 and check it.
・・・・・・It's been created, it's wonderful. . .
■Summary
What did you think?
It's a little complicated because you have to write code, but
it's convenient and interesting because you can do this without needing EC2.
This time I just touched on it briefly, but I'd like to try something a little more advanced.
That's all, thank you very much.