Monitor specific strings with CloudWatch Logs!

*Noodle restaurant Nishimura (Tsurumi Ward, Osaka City)

Hello!
My name is Hide, the ramen king of Beyond Osaka Office.
This is my 6th post.

Last time, I wrote about sending Cloudwatch alarms to Slack using Chatbot.
I wrote an interesting article about notifications using a useful tool called Chatbot, so if you're interested, please take a look!

Notify CloudWatch alarms to Slack with AWS Chatbot!

What is string monitoring in CloudWatch Logs?

CloudWatch Logs is a part of CloudWatch, which monitors AWS resources, and allows you to easily collect application and access logs from Amazon EC2 and save them on the cloud.

Since it can be saved on the cloud, you can easily search for specific logs when you want to check them.
Another attractive feature is that the collected logs can be managed centrally with one service, so management does not require much effort.

Next, character string monitoring involves monitoring application and access logs collected in real time to improve business efficiency, and if a certain output format or specific character string is detected, communication Refers to the monitoring method that notifies the tool. Even in the unlikely event that a problem occurs, it can be detected early, allowing you to quickly resolve the problem.

Implementing log monitoring with CloudWatch Logs is very easy, so let's walk through the steps together!

Configuration diagram

Install the agent used for CloudWatch Logs in EC2, and the agent will send the logs existing in EC2 to the console.

Next, alerts will be sent according to conditions that include specific strings set in the console. I don't think it's that difficult when you look at the configuration, so let's do our best to set it up.

Construction steps

① Create an IAM role

①-①: Go to IAM > Roles and click [Create role]

 

①-②: Select the following and click [Next]

● Trusted entity type: AWS service
● Use case: EC2

①-③: Select [CloudWatchFullAccess] and click [Next]

①-④: Fill in the role name and description and click [Create role]

 

①-⑤: EC2 > Select the created instance > Action > Security > Click Change IAM role

①-⑥: Select the IAM role you created earlier and click [Update IAM role]

*The application is complete if the security tab of the created instance shows the following status.

② CloudWatch Logs agent settings

②-①: Log in to the server

②-②: Install CloudWatch Logs agent

*For Amazon Linux, you can install with the following command.

yum install amazon-cloudwatch-agent

*The installation method is different for OSes other than Amazon Linux.
In the case of CentOS, the installation method is described with reference to the official document

1. Download the CloudWatch agent

*The download link differs depending on the OS, so please refer here.

wget https://s3.amazonaws.com/amazoncloudwatch-agent/centos/amd64/latest/amazon-cloudwatch-agent.rpm

2. Install CloudWatch agent

rpm -U ./amazon-cloudwatch-agent.rpm

*Also, the following agents deprecated , so please install them using the steps above.

yum install awslogs

②-③: Start the setup wizard

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

②-④: Configure agent settings

================================================== ============== = Welcome to the Amazon CloudWatch Agent Configuration Manager = = = = CloudWatch Agent allows you to collect metrics and logs from = = your host and send them to CloudWatch. Additional CloudWatch = = charges may apply. = ============================================ ==================== On which OS are you planning to use the agent? 1. linux 2. windows 3. darwin default choice: [1]:*Choose 1 for Linux 1 Trying to fetch the default region based on ec2 metadata... Are you using EC2 or On-Premises hosts? ) 1. EC2 2. On-Premises default choice: [1]: *Choose 1 for EC2 1 Which user are you planning to run the agent? Do you want to turn on StatsD daemon? (StatsD ) 1. yes 2. no default choice: [1]: *Required to collect custom metrics, so select 1 Which port do you want StatsD daemon to listen to ? (Which port does the StatsD daemon listen on? ) default choice: [8125] *Click Enter if there is no problem. What is the collect interval for StatsD daemon? 1. 10s 2. 30s 3. 60s default choice: [1]: *Please select 1 if there is no problem 1 What is the aggregation interval for metrics collected by StatsD daemon? 1. Do not aggregate 2. 10s 3. 30s 4. 60s default choice: [4]: ​​*Choose 4 if you are OK with it 4 Do you want to monitor metrics from CollectD? WARNING: CollectD must be installed or the Agent will fail to start? Do you want to monitor? Warning: If CollectD is not installed, the Agent will fail to start) 1. yes 2. no default choice: [1]: *Choose 2 as we will only use cloudwatchlogs 2 Do you want to monitor any host metrics? eg CPU, memory, etc. 1. yes 2. no default choice: [1]: *Since we only use cloudwatchlogs Select 2 2 Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration? Is there a configuration file for an existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) to use? ) 1. yes 2. no default choice: [2]: *There is no configuration file for cloudwatchlogs, so choose 2 2 Do you want to monitor any log files? 1 . yes 2. no default choice: [1]: *Please select 1 as the cloudwatchlogs configuration file is required 1 Log file path: *Please enter the path of the log you want to specify /var/log/messages Log group name: *Enter the log group name you want to specify default choice: [messages] cloudwatchlogs-test Log stream name: *Enter the log stream name you want to specify default choice: [{instance_id}] /var/log/ messages Log Group Retention in days 1. -1 2. 1 3. 3 4. 5 5. 7 6. 14 7. 30 8. 60 9. 90 10. 120 11. 150 12. 180 13. 365 14. 400 15. 545 16. 731 17. 1827 18. 2192 19. 2557 20. 2922 21. 3288 22. 3653 default choice: [1]: *Please enter the log retention period you want to specify ( -1 means unlimited) 1 Do you want to specify any additional log files to monitor? 1. yes 2. no default choice: [1]: *Settings Select 2 if you do not want to add the file 2 Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully. Current config as follows: { "agent": { "run_as_user": "root" }, "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/var/log/messages", "log_group_name": "cloudwatchlogs-test", " log_stream_name": "/var/log/messages", "retention_in_days": -1 } ] } } }, "metrics": { "metrics_collected": { "statsd": { "metrics_aggregation_interval": 60, "metrics_collection_interval": 10 , "service_address": ":8125" } } } } Please check the above content of the config. The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json. Edit it manually if needed. Do you want to store the config in the SSM parameter store? ) 1. yes 2. no default choice: [1]: *Please select 2 if you do not need to register with SSM 2 Program exits now.

*Supplementary materials

● Reference: StatsD ?
● Reference: CollectD ?

②-⑤: Start the agent

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

②-⑥: Check the agent status

systemctl status amazon-cloudwatch-agent

 

● amazon-cloudwatch-agent.service - Amazon CloudWatch Agent Loaded: loaded (/etc/systemd/system/amazon-cloudwatch-agent.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-08- 05 06:33:34 UTC; 3min 46s ago Main PID: 1352 (amazon-cloudwat) CGroup: /system.slice/amazon-cloudwatch-agent.service mq1352 /opt/aws/amazon-cloudwatch-agent/bin/amazon- cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json -pidfile /opt/ aws... Aug 05 06:33:34 ip-172-31-39-31.ap-northeast-1.compute.internal systemd[1]: Started Amazon CloudWatch Agent. Aug 05 06:33:34 ip-172 -31-39-31.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1352]: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it. Aug 05 06:33:34 ip-172-31-39-31.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1352]: I! Detecting run_as_user.. .

③ Check if it is collected on the console

③-①: Check and click the log group specified in cloudwatch > Log groups

③-②: Confirm that the log is saved with the specified log stream name and click

③-③: Check the log to see if it is being collected correctly

④ Create a metric filter

④-① Click [Create metric filter]

④-②: Define filter pattern

*There are various ways to define patterns, so please refer here

④-③: Test the filter pattern and if there are no problems, click [Next]

*Test method
1. Select [Custom log] or [Log stream name] from Select log data to test.

2.Create a log that matches the filter pattern in [Log event messages] and click [Test pattern]

3. Once the test results are reflected, the process is complete.

④-④: Fill in the information below.

● Filter name: The name of the filter itself
● Metric namespace: The name of the namespace to which CloudWatch sends metrics
● Metric name: The name of the metric created under the specified namespace
● Metric value: The name of the metric that will be published as a metric Numeric value
● Default value: value published to the metric when the pattern does not match, no value is published if left blank

④-⑤: If there are no problems, click [Create Metric Filter]

④-⑥: Completed when created as shown below

⑤ Set up SNS

⑤-①: Click [Create topic] in SNS > Topic

⑤-②: Set the following information and click [Create topic]

● Type: Standard
● Name/Description: Please specify freely

⑤-③: Click [Create subscription]

⑤-④: Fill in the following information and click [Create subscription]

● Protocol: Email
● Endpoint: Your email address

⑤-⑤: You will receive a confirmation email, so click [Confirm subscription]

⑤-⑥: Completed when the following message appears

⑥ Create a CloudWatch alarm

⑥-①: Output a log containing a specific string to the specified log file on the server as a test.

Since *messages is specified, the logger command is used, but any command that can output logs will do.
*We are testing because if a metric cannot be detected by the metric filter even once, you will not be able to select the metric when creating an alarm.

logger "ERROR"

⑥-②: Click [Create alarm] in CloudWatch > Alarm

⑥-③: Created namespace > Dimensionless metric > Select the created metric name and click [Select Metric]

⑥-④: Set the conditions and click [Next]

*Below, the condition is that if a specific character string is detected one or more times, an alarm will be notified.

⑥-⑤: Specify the topic you created and click [Next]

*Recovery notification settings are not required as it is character string monitoring.

⑥-⑥: Enter the created alarm name and click [Next]

⑥-⑦: If there are no problems with the settings, click [Create alarm]

⑦ Operation confirmation

⑦-①: Output a log containing a specific string to the specified log file on the server as a test.

Since *messages is specified, the looger command is used, but anything that can output logs will do.

logger "ERROR"

⑦-②: Check that the alarm is in [alarm state]

⑦-③: Check whether you have received an email regarding the alarm.

*If the alarm email is delivered to your email address, the process is complete.

summary

What do you think? I don't think it was that difficult to set a specific string in a metric filter to notify the logs collected by CloudWatch Logs as an alarm.

When monitoring strings on AWS, you can easily use CloudWatch, so please try using it for operational monitoring!

In addition to email, notifications can also be sent to Slack, Teams, and Chatwork. I wrote about how to set it up in my previous blog, so please check it out if you are interested!

thank you very much!

◇ I tried notifying CloudWatch alarm to Teams/Chatwork!

◇ Notify CloudWatch alarms to Slack with AWS Chatbot!