Learning from simulated failures! Access log analysis and Linux disk space investigation commands

2026.07.07

• Technical Blog
• Infrastructure

3

14

Introduction

Nice to meet you! I'm Ebi🦐, an infrastructure engineer who graduated in 2026!

As part of the practical training for infrastructure engineers, we conducted a simulated failure training exercise where senior employees set up a "simulated failure (server attack)" for the new employees to deal with

This time, we'll give a technical explanation of the two simulated failures that actually hit our servers, and how we investigated and identified the causes, starting with alerts from the integrated monitoring tool "Zabbix" 👓!

Current server configuration

The server configuration for this project is as follows:

The problem this time affectedour web serversone of

Two "pseudo-outages" that hit the server

During this training session, the following two attacks were launched 💀

1. Brute-force directory scraping is a web attack that attempts to extract information from a web server by brute-forcing its way through the paths of non-public files and directories. In this case, it caused a sharp increase in the CPU load on the web server.
2. Disk space depletion (mass generation of dummy files): This attack involves artificially generating large amounts of dummy files in various directories within the server, depleting the available disk space and causing the system to malfunction.
   These are all practical attacks that are highly likely to be encountered in actual operation and maintenance!

Zabbix, a monitoring tool that detects anomalies

What is Zabbix?

Zabbix is ​​the world's leading open-source (OSS) integrated monitoring software that monitors the status of servers, networks, and applications24/7, automating fault detection and alert notifications.

Reference: Zabbix official website (https://www.zabbix.com/jp)

[Main Features]

• Liveness monitoring and resource monitoring:This includes checking response times via Ping, measuring CPU usage, memory usage, remaining disk space, etc.
• Fault detection: Thesystem automatically detects abnormalities when pre-configured "thresholds" (such as CPU load "5" or disk usage "80%" in this example) are exceeded.
• Alert notifications: When a problem occurs , the responsible person will be immediately notified via email or chat tool (Chatwork, Slack, etc.)
• Visualization (Graphing):Collected data can be graphed in real time on a browser and centrally managed on a dashboard.

[Advantages and disadvantages]

• Benefits:It is completelyfree to use (open source)while offering advanced monitoring and scalability comparable to commercial tools.
• Disadvantages:the need to build the system from scratch, andits overly flexible configuration and unique GUI compared to SaaS solutions like Datadog. The learning curve for mastering its unique concepts is steep, and threshold tuning must be done entirely manually, resulting in increased operational workload…💦

The link below provides a more detailed explanation, so please take a look!

Reference: What is Zabbix? Basic concepts of monitoring and benefits of implementation (https://qiita.com/minesyouto8833/items/68f2c39eafc8a8bd2f47)

This alert notification

In this instance, it was the integrated monitoring tool "Zabbix" that quickly detected these anomalies and notified us. When the failure occurred, we received the following alert notification from Zabbix:

① Web server alert (high CPU load): PROBLEM:

• Current value: 24.185 (Severity: Warning)

② Jump server alert (disk pressure): PROBLEM:

• Current value: 82.09% (Severity: Warning)

Command log explanation used to identify and resolve the attack method

From here, we will explain the commands we actually used to identify the cause and how we interpreted the logs

 "Analyzing access logs" to uncover traces of attacks

 To identify the cause of a sudden surge in CPU load on a web server, it's crucial to cross-check three things in the access logs: "high-volume access IPs," "request paths," and "HTTP status codes." This is because external web attacks always leave traces in the web server's access logs, so when you notice an abnormal surge in traffic, you can identify the attack method by looking at the raw log data!

The commands used this timethe grep commandandthe awk command.

The grep commandis a powerful Linux command that searches for and extracts lines that match a specified string or regular expression from the output of a file or command.

The awk commandis a text processing command that reads text data and log files line by line and efficiently extracts lines that match specified conditions, as well as processes and aggregates data column by column (field).

By combining these two features, we've made it possible to display only the access logs from the time the alert occurred and only the information you want to see

The following is an example of its use! It was used to display the number of accesses per minute from the corresponding IP address at the time the alert occurred!

# Extract "alert occurrence time" from the logs and sort by the number of occurrences for each time period in descending order
$ grep -E "alert occurrence time" /var/log/httpd/xxxxxx | awk '{print $1}' | sort | uniq -c | sort -nr

The link below provides a more detailed and easy-to-understand explanation, so please take a look!

Reference: How to retrieve desired information from log files using grep and awk commands! (https://cgs580.hatenablog.com/entry/2023/12/03/155302)

The following is an excerpt from the access logs that we actually reviewed

▼ Number of accesses per minute from the relevant IP address at the time the alert occurred (※The number on the far left is the number of accesses) 383 54.250.xx.xx ★Japan 292 47.128.xx.xx ★Singapore ▼ Paths accessed by the relevant IP address and request status (※The number on the far left is the number of accesses) 10 54.250.xx.xx/swagger-ui.html 404 9 54.250.xx.xx/www.zip 404 8 54.74.xx.xx/.env.production 404

A large volume of access attempts targeting important configuration files such as /swagger-ui.html and /.env.production were detected in a short period of time from multiple IP addresses in Japan, Singapore, the United States, and other locations . Fortunately, all request statuses were "404 (Not Found) ," indicating that the web server correctly returned that the files did not exist. Therefore, we have determined that there has been no data leakage due to the attack at this time. Furthermore, since access from the attacking IPs has stopped, the CPU load has returned to normal levels.

Thus, when the CPU is under heavy load, simply restarting the server is not enough. Analyzing the access logs to determine "who (IP address)," "where (path)," and "what happened (e.g., 404 error)" is crucial for accurately understanding the situation of a web attack!

 Linux disk space check command to find internal anomalies

To resolve disk space pressure alerts, it's appropriate to use the `df` command to get an overall picture and then use the `du` and `ls` commands to narrow down the directory hierarchy. Zabbix alerts alone don't tell you "which files are consuming the most space," so you need to use OS-level (Linux) commands to identify abnormally large directories and hidden files.

The actual investigation process was as follows:

# 1. Check the file system type and overall disk usage
$ df -hT
# → Confirm that the root directory (/) is using 82%

# 2. Exclude other file systems and sort the root directory by size in descending order
$ du -hx --max-depth=1 / | sort -hr
# → Identify that /root and others are unusually large

# 3. Narrow down the target directory (/root) by sorting it further in descending order
$ du -hx --max-depth=1 /root | sort -hr

# 4. Identify all files, including hidden files, down to their individual sizes
$ du -hax --max-depth=1 /root | sort -hr

# 5. Check detailed information with the ls command
$ ls -laH /root

# 6. Display the "Last Modified Date" along with the size to identify when the dummy file was created
$ du -hax --time --max-depth=1 /root | sort -hr

This detailed investigation uncovered approximately 14GB of suspicious dummy files! 👀!

• /opt/dummy_data.bin (3.5G)
• /home/.dummy_backup.tar (3.5G)
• /var/log/dummy_app.log.bak (4.0G)
• /tmp/.dummy_cache.dat (3.5G)

This server only has 20GB of storage, so it's pretty much completely full... (;'∀')

Upon inspection, we found that all the files were empty (containing zeros), and it turned out they were files planted by a senior colleague to simulate a failure. We promptly deleted (cleaned up) these files and restored the disk space to its original safe state

Let's learn more about the `du` command!

`du` (disk usage) commandis used in UNIX-like operating systems such as Linux and macOS to check the disk space occupied by files and directories.

The following is a list of the main options for the du (disk usage) command

I've compiled a list of things I use frequently!

 Most commonly used basic options

 After completing the training

Through this simulated failure response, I learned firsthand the importance of "log analysis skills" and "OS internal investigation skills (how to use Linux commands)" that are essential for infrastructure engineers

When a problem occurs, it's easy to panic for a moment, but by using the alerts issued by Zabbix to logically unravel the cause one step at a time, you can safely and reliably restore the system!

Based on the experience and skills I gained during this training, I will continue to dedicate myself to my daily work with the goal of becoming a reliable engineer who can calmly handle any problems that may arise in the future!

Thank you for reading to the end <(_ _)>

If you found this article helpful,please give it a "Like"!

3

Loading...

3

14