[Micro Hardening v2] A simple security incident experience

Hello, this is Inusuki from the System Solutions Department

Security incidents are something that users are familiar with. One day, a service they use may become the target of a cyber attack, causing the service to become unavailable or for some functions to be suspended and the service to operate in a degraded state

This time, I participated in an event called "Micro Hardening," where you could actually experience such a security incident, and I have summarized my experience in a blog post

Refusal

・In order to avoid spoilers, this blog does not include actual examples of attacks carried out using Micro Hardening or detailed countermeasures.
・It contains a large amount of Inusuki's personal views.
・There are a total of six sets of exercises, but we will provide a digest version.

What is Micro Hardening?

Hardening is one of the sub-projects. Teams of 1-4 people work on a small cloud environment, protecting it from various cyber attacks and honing their ability to respond when attacked. The same attack is carried out in multiple sets over a short period of time

The original Hardening Project has high levels of required skill and high participation hurdles, so Micro Hardening, which anyone can easily participate in even if they have no experience in incident response, is a recommended hardening event for non-engineers and those just starting to learn about security

Differences between versions

Micro Hardening v2 has a higher number of incidents than v1

SLA (Service Level Agreement) has been introduced as a standard for meeting business requirements

Main Story

introduction

To get to Awaji Island, where the competition venue is located, I took the subway, JR, and bus to get there
The day beforeFeared by numerous incidentsI was so excited that I couldn't sleep, so I took a nap on the bus after my connecting flight

That aside, Awaji Island
is a popular tourist destination, and its location is excellent, with a panoramic view of the Seto Inland Sea even available from the competition venue, SAKIA.

SAKIA and includes a restaurant, hotel, co-working space, and a friendly atmosphere.

Pre-briefing and team division

Based on the advance materials, there was an explanation of the event and team presentations.
(The materials mainly focused on the competition environment, so I will not go into detail here.)

The team composition was announced on the day, so we were unable to communicate in advance, but Team 1, which Inusuki was on, consisted of three programmers and Inusuki

It was interesting to talk to everyone because each person had a different background and reason for participating in the event

Event Start

First set

"GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__aaaaaaa___________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a____a_a____aa___aa__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_________a___a___a_a_a_a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a__aaaa__a_____a__a__a__a__aaaaa_____________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a__aaaaaaa__a_____a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a__a_____a__a_____a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa___a_____a__a_____a__aaaaaaa___________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa___aaaaaaa_____a_____aaaaaa___aaaaaaa__ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a_____a_______a_a____a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a___________a______a___a___a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__aaaaaa______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /__________a_____a_____aaaaaaa__a___a_______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a_____a_____a_____a__a____a______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0"

The ASCII art of [GAME START] is displayed in the access log when the game starts, and so Micro Hardening has begun.
For the time being, we decided to go in with no precautions, focusing on just observing, organizing the events, and creating a timeline.

A few minutes had passed since the start.
No incidents of any particular relevance had occurred, and the tension within him began to subside.
(Inusuki was unaware at the time that this was just the calm before the storm.)

A short time later, the team received a report that the site title had been altered...
When they reloaded the e-commerce site, the words "hacked anonymous" appeared in the site title.
What a bold declaration of war!

First, I investigated the access logs for the relevant period and found a request to a suspicious URI.
Moreover, it returned an HTTP status code of 200.
When I looked under the document root, I found an unfamiliar PHP file with a common file name.
It seems that some kind of vulnerability had been exploited to install a backdoor.

Meanwhile, an incident occurs in which product prices on an e-commerce site are fraudulently manipulated.
It is likely that administrative user information for the e-commerce site's database was leaked, or that records were fraudulently updated via SQL injection.
If it's the former, it may be too late to find out what happened at this point.

At the same time, login forms that were misleading to users were embedded on the product pages, and the page layout was changed to suit the attackers' preferences, making the page look dreadful...
and in the end, the service went down, and the first set came to an end in a terrible state. It was
really a terrible experience. It was fun.

The results of the first set were stagnant for all teams.

The defense column increases or decreases depending on how many evaluation items are cleared.
This time there was no defense, so the team's defense score was 0.
However, if the defense is too rigid, it may affect the purchase step of the legitimate crawler, and it seems that defense does not necessarily improve the SLA.

〇
In the second and third sets, which continued, we were unable to identify a specific intrusion route or implement fundamental countermeasures, so our strategy was to split up and endure without interrupting the service, and we made various setting adjustments and applied security patches during the preparation period.

As mentioned above, attacks and incidents occur at specific times according to a timeline, so once you have taken measures, you will likely feel relatively at ease in the future

However, the service outages continued.
However, we were able to temporarily prevent some of the intrusions and tampering.

〇 Sets 4 and 5
Based on the above results, in sets 4 and 5 we decided to review areas for improvement and devise further improvement and mitigation measures based on the vulnerabilities and incidents that had surfaced by set 3.

In particular, it would be fatal if the service were to stop at some point, so this must be avoided.
Because a timeout was returned before the event reached the web server, we thought that there might be some kind of problem occurring at a different layer.

Because the competitive environment involves a mixture of multiple servers, it would have been better for the team to have a detailed understanding of these specifications and organize them in order...
In the end, the improvements and mitigation measures we implemented in this set were not very effective, and we were unable to avoid a decline in our SLA...

of the 6th set,
we received some hints, and decided to check again based on those hints.
As a result, we were able to improve the situation by simply restarting a certain service.

Even as I write this blog, I deeply regret that I was so preoccupied with responding to other incidents that I neglected to thoroughly examine the service logs, especially since it was a service I had my eye on.
However, it appears that the measures mentioned above were well-received, and the SLA continued to improve without any stagnation.

The results, including the final set, are as follows.


As it is said that you can only perform at 50% at the beginning, we can see that the difference in scores between the first and sixth sets was more than double for each team.
This shows the importance of repeated practice.

In fact, I think the team was able to respond smoothly to incidents in the first and sixth sets

Tips for enjoying Micro Hardening to the fullest

Below are some tips (including Inusuki's reflections) for those planning to attend future events to help you enjoy Micro Hardening 100%

1. Some prior knowledge is required.
If you are a non-engineer or a beginner, it is recommended that you understand how to use the commands that will be tested in the LPIC-1 exam.

The exercises cover basic specifications and provide additional information on how to restart services, but do not go so far as to explain commands

2. Team building is extremely important.
Let's work out the purpose of the set, who will play what role, and how we will approach the next set.

In this case, we only created a timeline in the first set, and didn't take the time to understand and organize the specifications within the team. This had a big impact later on, and we realized that this was a problem. Even if each person understands the specifications, it is important to look them over as a team and align your understanding

3. Leave a trail of your work.
This can be done in Slack or a Google spreadsheet, so be sure to leave a trail. This
is also important for organizing who did what later on.

4. Consider using external services and tools.
In order to expose problems and maintain service quality, it is more realistic to make full use of external services and tools.

Don't think you can do it on your own like Inusuki did.
The teams that are ranked high in the rankings are good at using this area.

5. Be mindful of meeting business requirements Simply
having a service running will not improve your score.
Think about how you can improve your score and what it means to meet business requirements.

summary

It may be difficult to immediately implement a competition-level environment, but I would like to at least hold a security incident experience session within the company

Thank you for reading this far!