Try Amazon CloudWatch Logs

This is Sashihara, an infrastructure engineer!

AWS has a service called CloudWatch that monitors things like CPU usage.
CloudWatch has various functions, but this time I tried out CloudWatch Logs, which I was interested in.

What is CloudWatch Logs?

This is a service that allows you to monitor log files of EC2 and other applications, and generate alarms when specific character strings are confirmed.

Monitoring log files - Amazon CloudWatch

For now, I will try sending the access log of apache on EC2.

Creating an IAM role

Create an IAM role to send log files to CloudWatch Logs.

Since log files will be sent from EC2, select Amazon EC2 as the role type.

A CloudWatch Logs policy is available, so we will use it this time.

Select CloudWatchlogsFullAccess

This is all you need to configure IAM.

After that, create an EC2 instance with an IAM role assigned.

Installing awslogs

After logging into the launched instance, install the dedicated agent awslogs.

[ec2-user@ip-172-xxx-xxx-xxx ~]$ sudo yum install awslog

Next, change the configuration file.
The default settings use CloudWatch in the Northern Virginia region (us-east-1), so change it to the Tokyo region (ap-northeast-1).

[ec2-user@ip-172-xxx-xxx-xxx ~]$ sudo vim /etc/awslogs/awscli.conf [default] region = us-east-1 ⇒region = ap-northeast-1

Start the agent and configure automatic startup settings.

[ec2-user@ip-172-xxx-xxx-xxx ~]$ sudo /etc/init.d/awslogs start Starting awslogs: [ OK ]

Auto start settings

[ec2-user@ip-172-xxx-xxx-xxx ~]$ sudo chkconfig awslogs on

It should now be sent!
Let's check it out! !

Confirm log sending

If you check on the console, you will see that a log group called "/var/log/messages" has been created.

When you click it, the instance ID is output to the log stream.

Furthermore, if you click on this, you can check the contents of the messages.

It went well!

Why are messages written to CloudWatch Logs?

⇒This is because messages is set by default.

The configuration is written in /etc/awslogs/awslogs.conf.

[/var/log/messages] datetime_format = %b %d %H:%M:%S file = /var/log/messages buffer_duration = 5000 log_stream_name = {instance_id} initial_position = start_of_file log_group_name = /var/log/messages

Send apache access log

Now let's output apache access logs to CloudWatch Logs!

Modify the configuration file on the server.

[ec2-user@ip-172-xxx-xxx-xxx ~]$ sudo vim /etc/awslogs/awslogs.conf

Add the following.

[/var/log/httpd/] file = /var/log/httpd/access_log buffer_duration = 5000 log_stream_name = {hostname} initial_position = start_of_file log_group_name = /var/log/httpd

This is the content of the above settings.

file

Specify the log file to be pushed to CloudWatch Logs (wildcard specifications such as /var/log/httpd/* are also possible.)

buffer_duration

Specify the batch period of log events (5000 is the minimum value and default)

log_stream_name

Log stream settings (default is instance_id, but this time I will use hostname)

initial_position

There is also end_of_file to specify the data read position, but I think the default start_of_file is basically fine.

log_group_name

Specify the destination log group.

Restart awslogs for the settings to take effect.

[ec2-user@ip-172-xxx-xxx-xxx ~]$ sudo /etc/init.d/awslogs restart

Confirm log sending

Let's check this out! !
Check again from the console. . .

"/var/log/httpd" has been added! !
Click further. .

A log stream is created with the host name! !

I was also able to check the apache access log! !

That was easy! !
This time we just sent the apache access log, but it is also possible to monitor the HTTP status code and send an alarm when a 40x error occurs.

You can also perform log analysis in conjunction with ElasticSearch.

summary

• What is CloudWatch Logs? A log collection service
• An agent called awslogs is convenient to use.
• It's super easy to just send

Next, let's try implementing a serverless architecture using AWS Lambda, which is a hot topic!