Get access logs with AWS ELB!

＊Chicken Soba Zagin Main Store (Higobashi, Osaka City)

 

Hello!
My name is Hide, the ramen king of Beyond Osaka Office.
This is my second post.
This time, I would like to write about how to obtain access logs with ELB.
Last time, let's have fun using docker commands with Makefile! ] I also wrote an article,
so if you use docker, please take a look.

Have fun using docker commands with Makefile!

 

 

What are the benefits of acquiring access logs with ELB?

Some of you reading this article may be thinking, ``Huh?
It's not necessary because

It is certainly possible to obtain access logs on the server, but
there may not be much benefit in obtaining access logs on ELB....
As the poster, I thought the same thing, ``I can just use the server, so there's no benefit...''

 

 

But what if you want to analyze access logs?
If the servers are distributed across multiple servers, access logs will be saved on each server.

Therefore, even if you try to analyze the access logs, since the access logs exist on different servers, it
will take a lot of time and effort to integrate and analyze the access logs.

For example, you can use CloudWatchLogs to aggregate logs.
In that case, you will need to manage permissions with IAM, install an agent on the target server, and
edit settings in awslogs.conf....

You can also easily collect and utilize logs
using an open source data log collection tool called fluentd However, this also requires installing and configuring tools through permission management with IAM.
Furthermore, it takes a lot of time to specify S3 as the aggregation destination...

 

 

you can easily aggregate access logs
by enabling the ELB access logs in front of the server without using CloudWatchLogs or fluentd. !

All access logs are aggregated in S3, making storage and analysis easy.
Furthermore, by using AWS Athena and Redshift,
easily store data in Amazon S3 using standard SQL. You can analyze it!

Additionally, even if the access log is lost due to a hang-up,
you can easily check what happened by checking the ELB access log.

 

And since ELB is a fully managed AWS service, you can use it with confidence.
Even if something goes wrong, AWS will take responsibility for fixing it.
that's why! !
Please rest assured that there is almost no chance that you will not be able to obtain an access log even though it has been activated.

 

Now you know the benefits of enabling access logging in ELB!
Now, let's take a look at the setup steps together!

 

 

Setting procedure

①Move to EC2⇛load balancer

 

 

 

②Click [Edit Attributes] from Actions

 

 

 

③ Select enable access log

 

 

 

④Specify the bucket name for the S3 location

 

 

⚠① S3 buckets can only contain combinations of lowercase letters, numbers, periods (.), and dashes (-).
Please note that if you enter anything other than the above, an error will occur as shown in the image below.

 

 

 

⚠② If the S3 bucket name you entered is already in use, a warning will be displayed, so please enter another name.

 

⑤ Enter ✔ in [Create this location] and click Save.

 

 

⑥Move to S3 and click [S3 bucket name you created]

 

 

 

 

⑦If you can confirm that there is an access log file in the following directory, the process is complete.

*Log files are created every 5 minutes.

/Created S3 bucket name/AWSLogs/Elastic Load Balancing account ID number/elasticloadbalancing/Region name/Creation year later/Creation month/Creation date/

 

 

 

 

 

How to check access log

① Go to IAM and click [Create user]

 

 

② Fill in the required information

*Be sure to check [Programmatic access] under Access type.

 

 

 

③ Select the [AmazonS3ReadOnlyAccess] policy

 

 

 

 

④ Check if there are any errors in the input and create a user

 

 

 

⑤Download csv

*Just in case, make a note of your access key and secret access key.

 

 

⑥Install AWS CLI

*For information on installing AWS CLI, please refer to the AWS official documentation.

⚠Here, we are using Centos7.
For Amazon Linux2, installation is not necessary as it is already installed.

 

⑥-① Make sure that Python is version 2.7 or later
*If you do not have Python, please install it separately.
*2.7.5 is installed by default on Centos7.

python --version Python 2.7.5

 

 

⑥-② Install the bundled installer

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" sudo yum install -y unzip unzip awscli-bundle.zip sudo ./awscli-bundle/ install -i /usr/local/aws -b /usr/local/bin/aws aws --version

 

 

⑦AWS CLI settings

aws configure AWS Access Key ID [None]: Enter the IAM access key you created AWS Secret Access Key [None]: Enter the IAM secret key you created Default region name [None]: Enter the region Default output format [None] :Enter the file format you want to output

 

 

⑧Output the access log

 
*List files in S3 bucket with AWS CLI S3 command aws s3 ls s3://tokuhara-test-alb-access-log/AWSLogs/485076298277/elasticloadbalancing/ap-northeast-1/2021/01/18/ 2021- 01-18 08:05:12 884 485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T0805Z_54.65.29.6_53o3j7p0.log.gz *Copy locally aws s3 cp s3://tokuhara-test-alb-access- log/AWSLogs/485076298277/elasticloadbalancing/ap-northeast-1/2021/01/18/485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68. 106.116_5qgag1lu.log.gz ./ *Can be copied locally ls -l ./ -rw-r--r--. 1 root root 295 Jan 18 23:55 485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68.106.11 6_5qgag1lu.log.gz *Compressed Unzip the file gunzip 485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68.106.116_5qgag1lu.log.gz *Show access log less 485076298277_ elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68.106.116_5qgag1lu. log http 2021-01-21T00:10:51.821814Z app/tokuhara-test-alb/20a1e21abd53f9e7 63.143.XX.XX:45662 - -1 -1 -1 503 - 110 337 "GET http://example.com:80 / HTTP/1.1" "Go-http-client/1.1" - - arn:aws:elasticloadbalancing:ap-northeast-1:485076298277:targetgroup/tokuahra-test-tg-ec2/ee1e50320f3398ec "Root=1-6008c68b-4198b51b343e8f127df30e01 " "-" "-" 0 2021-01-21T00:10:51.821000Z "forward" "-" "-" "-" "-" "-" "-"                                                                                                                                                                                                                                                                                                      

 

 

 

Output results and format

ELB access log settings! ! Thank you for your hard work٩(ˊᗜˋ*)و
However, even if you set the access log to be output, if you don't know how to read it, you won't be able to investigate the cause....

 

that's why! While referring to the "AWS Official Documentation",

we have prepared the following format
for the sample access log obtained by ELB Please take a look and see what items are available!

 

http 2021-01-21T00:10:51.821814Z app/tokuhara-test-alb/20a1e21abd53f9e7 63.143.XX.XX:45662 - -1 -1 -1 503 - 110 337 "GET http://example.com:80/ HTTP/1.1" "Go-http-client/1.1" - - arn:aws:elasticloadbalancing:ap-northeast-1:485076298277:targetgroup/tokuahra-test-tg-ec2/ee1e50320f3398ec "Root=1-6008c68b-4198b51b343e8f127df30e01" " -" "-" 0 2021-01-21T00:10:51.821000Z "forward" "-" "-" "-" "-" "-" "-"

 

 

 

 

Fee

Regarding fees, ELB fees will be charged.
According to the "AWS official document" regarding ELB,
usage fees for Application Load Balancers in the AWS Tokyo region are as follows.

 

Next, after acquiring the access log, it is stored in S3, so S3 fees will be incurred.
According to the "AWS Official Documentation" regarding S3,
the pricing for S3 in the AWS Tokyo Region is as follows.

 

 

It's very difficult to understand, but up to the first 50 TB, you will be charged 0.023 USD per GB per month.

For example, if the log file accumulated per month is 10 GB, approximately 24.28 yen will be incurred in the first month,
20 GB will be accumulated in the second month, so approximately 48.56 yen will be incurred, and
30 GB will be incurred in the third month. Due to the accumulated amount, a fee of approximately 72.54 yen will be charged.
From the month in which the capacity exceeds 450TB, a fee of 0.022 USD per GB will be charged.

If you look at the fee calculations like this, you can see that access logs can be obtained at a very low cost.

 

 

 

summary

What do you think?
Wouldn't it be very easy to aggregate access logs?

By enabling ELB's access log to consolidate access logs, you can reduce time and effort, and
you can use it stably as it is almost never impossible to obtain logs due to a problem!
Even if something goes wrong with the server and a log defect occurs, you can easily check it! !

that's why! !
sure to enable ELB access logs so you can operate your server safely and conveniently.

Lastly, if you're reading this article and are thinking, "I'm starting to get interested in S3!!",
please check out the article below written by Persian Neko from Beyond. !!

[For AWS beginners] Easy to understand! What is Amazon S3?

 

Thank you for reading the article!