[Osaka/Yokohama/Tokushima] Looking for infrastructure/server side engineers!

[Osaka/Yokohama/Tokushima] Looking for infrastructure/server side engineers!

[Deployed by over 500 companies] AWS construction, operation, maintenance, and monitoring services

[Deployed by over 500 companies] AWS construction, operation, maintenance, and monitoring services

[Successor to CentOS] AlmaLinux OS server construction/migration service

[Successor to CentOS] AlmaLinux OS server construction/migration service

[For WordPress only] Cloud server “Web Speed”

[For WordPress only] Cloud server “Web Speed”

[Cheap] Website security automatic diagnosis “Quick Scanner”

[Cheap] Website security automatic diagnosis “Quick Scanner”

[Reservation system development] EDISONE customization development service

[Reservation system development] EDISONE customization development service

[Registration of 100 URLs is 0 yen] Website monitoring service “Appmill”

[Registration of 100 URLs is 0 yen] Website monitoring service “Appmill”

[Compatible with over 200 countries] Global eSIM “Beyond SIM”

[Compatible with over 200 countries] Global eSIM “Beyond SIM”

[If you are traveling, business trip, or stationed in China] Chinese SIM service “Choco SIM”

[If you are traveling, business trip, or stationed in China] Chinese SIM service “Choco SIM”

[Global exclusive service] Beyond's MSP in North America and China

[Global exclusive service] Beyond's MSP in North America and China

[YouTube] Beyond official channel “Biyomaru Channel”

[YouTube] Beyond official channel “Biyomaru Channel”

Get access logs with AWS ELB!

Chicken Soba Zagin Main Store (Higobashi, Osaka City)

 

Hello!
My name is Hide, the ramen king of Beyond Osaka Office.
This is my second post.
This time, I would like to write about how to obtain access logs with ELB.
Last time, let's have fun using docker commands with Makefile! ] I also wrote an article,
so if you use docker, please take a look.

Have fun using docker commands with Makefile!

 

 

What are the benefits of acquiring access logs with ELB?

Some of you reading this article may be thinking, ``Huh?
It's not necessary because

It is certainly possible to obtain access logs on the server, but
there may not be much benefit in obtaining access logs on ELB....
As the poster, I thought the same thing, ``I can just use the server, so there's no benefit...''

 

 

But what if you want to analyze access logs?
If the servers are distributed across multiple servers, access logs will be saved on each server.

Therefore, even if you try to analyze the access logs, since the access logs exist on different servers, it
will take a lot of time and effort to integrate and analyze the access logs.

For example, you can use CloudWatchLogs to aggregate logs.
In that case, you will need to manage permissions with IAM, install an agent on the target server, and
edit settings in awslogs.conf....


You can also easily collect and utilize logs
using an open source data log collection tool called fluentd However, this also requires installing and configuring tools through permission management with IAM.
Furthermore, it takes a lot of time to specify S3 as the aggregation destination...

 

 


you can easily aggregate access logs
by enabling the ELB access logs in front of the server without using CloudWatchLogs or fluentd. !

All access logs are aggregated in S3, making storage and analysis easy.
Furthermore, by using AWS Athena and Redshift,
easily store data in Amazon S3 using standard SQL. You can analyze it!

Additionally, even if the access log is lost due to a hang-up,
you can easily check what happened by checking the ELB access log.

 

And since ELB is a fully managed AWS service, you can use it with confidence.
Even if something goes wrong, AWS will take responsibility for fixing it.
that's why! !
Please rest assured that there is almost no chance that you will not be able to obtain an access log even though it has been activated.

 

Now you know the benefits of enabling access logging in ELB!
Now, let's take a look at the setup steps together!

 

 

Setting procedure

①Move to EC2⇛load balancer

 

 

 

②Click [Edit Attributes] from Actions

 

 

 

③ Select enable access log

 

 

 

④Specify the bucket name for the S3 location

 

 

⚠① S3 buckets can only contain combinations of lowercase letters, numbers, periods (.), and dashes (-).
Please note that if you enter anything other than the above, an error will occur as shown in the image below.

 

 

 

⚠② If the S3 bucket name you entered is already in use, a warning will be displayed, so please enter another name.

 

⑤ Enter ✔ in [Create this location] and click Save.

 

 

⑥Move to S3 and click [S3 bucket name you created]

 

 

 

 

⑦If you can confirm that there is an access log file in the following directory, the process is complete.

*Log files are created every 5 minutes.

/Created S3 bucket name/AWSLogs/Elastic Load Balancing account ID number/elasticloadbalancing/Region name/Creation year later/Creation month/Creation date/

 

 

 

 

 

How to check access log

① Go to IAM and click [Create user]

 

 

② Fill in the required information

*Be sure to check [Programmatic access] under Access type.

 

 

 

③ Select the [AmazonS3ReadOnlyAccess] policy

 

 

 

 

④ Check if there are any errors in the input and create a user

 

 

 

⑤Download csv

*Just in case, make a note of your access key and secret access key.

 

 

⑥Install AWS CLI

*For information on installing AWS CLI, please refer to the AWS official documentation.

⚠Here, we are using Centos7.
For Amazon Linux2, installation is not necessary as it is already installed.

 

⑥-① Make sure that Python is version 2.7 or later
*If you do not have Python, please install it separately.
*2.7.5 is installed by default on Centos7.

python --version Python 2.7.5

 

 

⑥-② Install the bundled installer

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" sudo yum install -y unzip unzip awscli-bundle.zip sudo ./awscli-bundle/ install -i /usr/local/aws -b /usr/local/bin/aws aws --version

 

 

⑦AWS CLI settings

aws configure AWS Access Key ID [None]: Enter the IAM access key you created AWS Secret Access Key [None]: Enter the IAM secret key you created Default region name [None]: Enter the region Default output format [None] :Enter the file format you want to output

 

 

⑧Output the access log

 
*List files in S3 bucket with AWS CLI S3 command aws s3 ls s3://tokuhara-test-alb-access-log/AWSLogs/485076298277/elasticloadbalancing/ap-northeast-1/2021/01/18/ 2021- 01-18 08:05:12 884 485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T0805Z_54.65.29.6_53o3j7p0.log.gz *Copy locally aws s3 cp s3://tokuhara-test-alb-access- log/AWSLogs/485076298277/elasticloadbalancing/ap-northeast-1/2021/01/18/485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68. 106.116_5qgag1lu.log.gz ./ *Can be copied locally ls -l ./ -rw-r--r--. 1 root root 295 Jan 18 23:55 485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68.106.11 6_5qgag1lu.log.gz *Compressed Unzip the file gunzip 485076298277_elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68.106.116_5qgag1lu.log.gz *Show access log less 485076298277_ elasticloadbalancing_ap-northeast-1_app.tokuhara-test-alb.20a1e21abd53f9e7_20210118T2355Z_52.68.106.116_5qgag1lu. log http 2021-01-21T00:10:51.821814Z app/tokuhara-test-alb/20a1e21abd53f9e7 63.143.XX.XX:45662 - -1 -1 -1 503 - 110 337 "GET http://example.com:80 / HTTP/1.1" "Go-http-client/1.1" - - arn:aws:elasticloadbalancing:ap-northeast-1:485076298277:targetgroup/tokuahra-test-tg-ec2/ee1e50320f3398ec "Root=1-6008c68b-4198b51b343e8f127df30e01 " "-" "-" 0 2021-01-21T00:10:51.821000Z "forward" "-" "-" "-" "-" "-" "-"                                                                                                                                                                                                                                                                                                      

 

 

 

Output results and format

ELB access log settings! ! Thank you for your hard work٩(ˊᗜˋ*)و
However, even if you set the access log to be output, if you don't know how to read it, you won't be able to investigate the cause....

 

that's why! While referring to the "AWS Official Documentation",

we have prepared the following format
for the sample access log obtained by ELB Please take a look and see what items are available!

 

http 2021-01-21T00:10:51.821814Z app/tokuhara-test-alb/20a1e21abd53f9e7 63.143.XX.XX:45662 - -1 -1 -1 503 - 110 337 "GET http://example.com:80/ HTTP/1.1" "Go-http-client/1.1" - - arn:aws:elasticloadbalancing:ap-northeast-1:485076298277:targetgroup/tokuahra-test-tg-ec2/ee1e50320f3398ec "Root=1-6008c68b-4198b51b343e8f127df30e01" " -" "-" 0 2021-01-21T00:10:51.821000Z "forward" "-" "-" "-" "-" "-" "-"

 

field explanation sample value
type Type of request or connection http
time The time (UTC) when the load balancer received the request from the client 2021-01-21T00:10:51.821814Z
elb ELB name app/tokuhara-test-alb/20a1e21abd53f9e7
client:port IP address and port of the client that sent the request 63.143.XX.XX:45662
target:port The IP address and port of the target that served this request. -
request_processing_time Total elapsed time from the time the ELB receives the request until it sends the request to the target -1
target_processing_time Total elapsed time from the time the ELB sends the request to the target to the time the target starts sending response headers -1
response_processing_time Total elapsed time from the time the ELB receives the response headers from the target to the time it starts sending the response to the client -1
elb_status_code Status code of response from load balancer 503
target_status_code Status code of response from target -
received_bytes Size of request received from client 110
sent_bytes Size of response returned to client 337
"request" The line is enclosed in double quotes in the request from the client "GET http://example.com:80/ HTTP/1.1"
"user_agent" User-Agent string identifying the requesting client "Go-http-client/1.1"
ssl_cipher Set to - if it is an SSL cipher and not HTTPS -
ssl_protocol If it is an SSL protocol and not HTTPS, it will be set to - -
target_group_arn Target group Amazon resource name arn:aws:elasticloadbalancing:ap-northeast-1:485076298277:targetgroup/tokuahra-test-tg-ec2/ee1e50320f3398ec
"trace_id" Contents of the X-Amzn-Trace-Id header (enclosed in double quotes). "Root=1-6008c68b-4198b51b343e8f127df30e01"
"domain_name" The SNI domain provided by the client during the TLS handshake (enclosed in double quotes), set to - if not HTTPS "-"
"chosen_cert_arn" ARN of the certificate presented to the client (enclosed in double quotes). If not HTTPS, it will be set to - "-"
matched_rule_priority Priority value of the rule that matched the request 0
request_creation_time The time the ELB received the request from the client 2021-01-21T00:10:51.821000Z
"actions_executed" Actions to be performed when processing a request (enclosed in double quotes) "forward"
"redirect_url" URL of the redirect target in the location header of the HTTP response (enclosed in double quotes) "-"
"error_reason" Error reason code (enclosed in double quotes) "-"
"target:port_list" A space-separated list of IP addresses and ports (enclosed in double quotes) of the targets that served this request. "-"
"target_status_code_list" Space-separated list of status codes from the target's response (enclosed in double quotes) "-"
"classification" desync mitigation classification (enclosed in double quotes) "-"
"classification_reason" Classification reason code (enclosed in double quotes). "-"

 

 

 

Fee

Regarding fees, ELB fees will be charged.
According to the "AWS official document" regarding ELB,
usage fees for Application Load Balancers in the AWS Tokyo region are as follows.

 

Next, after acquiring the access log, it is stored in S3, so S3 fees will be incurred.
According to the "AWS Official Documentation" regarding S3,
the pricing for S3 in the AWS Tokyo Region is as follows.

 

 

It's very difficult to understand, but up to the first 50 TB, you will be charged 0.023 USD per GB per month.

For example, if the log file accumulated per month is 10 GB, approximately 24.28 yen will be incurred in the first month,
20 GB will be accumulated in the second month, so approximately 48.56 yen will be incurred, and
30 GB will be incurred in the third month. Due to the accumulated amount, a fee of approximately 72.54 yen will be charged.
From the month in which the capacity exceeds 450TB, a fee of 0.022 USD per GB will be charged.

If you look at the fee calculations like this, you can see that access logs can be obtained at a very low cost.

 

 

 

summary

What do you think?
Wouldn't it be very easy to aggregate access logs?

By enabling ELB's access log to consolidate access logs, you can reduce time and effort, and
you can use it stably as it is almost never impossible to obtain logs due to a problem!
Even if something goes wrong with the server and a log defect occurs, you can easily check it! !

that's why! !
sure to enable ELB access logs so you can operate your server safely and conveniently.

Lastly, if you're reading this article and are thinking, "I'm starting to get interested in S3!!",
please check out the article below written by Persian Neko from Beyond. !!

[For AWS beginners] Easy to understand! What is Amazon S3?

 

Thank you for reading the article!

 

If you found this article helpful , please give it a like!
0
Loading...
0 votes, average: 0.00 / 10
22,130
X facebook Hatena Bookmark pocket
[2025.6.30 Amazon Linux 2 support ended] Amazon Linux server migration solution

[2025.6.30 Amazon Linux 2 support ended] Amazon Linux server migration solution

[Osaka/Yokohama] Actively recruiting infrastructure engineers and server side engineers!

[Osaka/Yokohama] Actively recruiting infrastructure engineers and server side engineers!

The person who wrote this article

About the author

Hide@Infrastructure Engineer

It all started with a very interesting interview.
A mid-career employee of the System Solutions Department in Osaka.My
job is to build and operate servers and clouds!
I have the qualifications of LPIC1, AWS SAA, and OCI Architect Associate.

Actually, I love ramen and
have investigated over 100 stores in Osaka (。-∀-) I'm striving to become the Ramen King of Nibi Beyond
!

I'm also on Twitter, so please follow me (´∇`)
Click on the Twitter mark on the right corner! !