[Micro Hardening v2] A simple security incident experience

Hello, this is Inusuki from the System Solutions Department

Security incidents are something that users are familiar with. One day, a service they use may become the target of a cyber attack, causing the service to become unavailable or for some functions to be suspended and the service to operate in a degraded state

This time, I participated in an event called "Micro Hardening," where you could actually experience such a security incident, and I have summarized my experience in a blog post

Refusal

- To avoid spoilers, this blog does not include specific examples of attacks performed in Micro Hardening or detailed countermeasures.
- This post contains a significant amount of Inusuki's personal opinion.
- There are a total of 6 sets of exercises, but this post will be a digest.

What is Micro Hardening?

Hardening is one of the sub-projects. Teams of 1-4 people work on a small cloud environment, protecting it from various cyber attacks and honing their ability to respond when attacked. The same attack is carried out in multiple sets over a short period of time

The original Hardening Project has high levels of required skill and high participation hurdles, so Micro Hardening, which anyone can easily participate in even if they have no experience in incident response, is a recommended hardening event for non-engineers and those just starting to learn about security

Differences between versions

Micro Hardening v2 has a higher number of incidents than v1

as a standard for meeting operational requirements SLA) has been newly introduced

Main Story

introduction

To get to Awaji Island, where the competition venue is located, I took the subway, JR, and bus to get there
The day beforeFeared by numerous incidentsI was so excited that I couldn't sleep, so I took a nap on the bus after my connecting flight

Putting that aside, Awaji Island.
As expected, being a popular tourist destination, the location is excellent, and you can even see the Seto Inland Sea from the competition venue, SAKIA.

Furthermore,SAKIA is a community facility created by renovating a closed elementary school, and it houses a restaurant, hotel, and co-working space, creating a friendly and welcoming atmosphere.

Pre-briefing and team division

Based on the pre-event materials, there was an explanation of the event and team presentations.
(The main content of the materials was information about the competition environment, so I will omit it here.)

The team composition was announced on the day, so we were unable to communicate in advance, but Team 1, which Inusuki was on, consisted of three programmers and Inusuki

It was interesting to talk to everyone because each person had a different background and reason for participating in the event

Event Start

First set

"GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__aaaaaaa___________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a____a_a____aa___aa__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_________a___a___a_a_a_a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a__aaaa__a_____a__a__a__a__aaaaa_____________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a__aaaaaaa__a_____a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a__a_____a__a_____a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa___a_____a__a_____a__aaaaaaa___________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa___aaaaaaa_____a_____aaaaaa___aaaaaaa__ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a_____a_______a_a____a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a___________a______a___a___a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__aaaaaa______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /__________a_____a_____aaaaaaa__a___a_______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a_____a_____a_____a__a____a______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0"

As soon as the game starts, the ASCII art "[GAME START]" appears in the access log, indicating that... Micro Hardening has begun.
For now, the plan is to simply observe, organize the events, and create a timeline, so we'll proceed without any defenses.

A few minutes had passed since the start.
No incidents of any sort had occurred, and inwardly, the tension was easing.
(Little did Inusuki know at the time that this was the calm before the storm.)

A little later, a report came in within the team that the site title had been tampered with...
When we reloaded the e-commerce site, the words "hacked anonymous" appeared in the site title.
What a bold declaration of war!

Upon investigating the access logs around the relevant period, we found requests to suspicious URIs. Moreover
, they were returning an HTTP status code of 200.
Looking under the document root, we found unfamiliar PHP files with common filenames.
It appears that some kind of vulnerability was exploited, and a backdoor was installed.

Meanwhile, an incident occurred where product prices on the e-commerce site were fraudulently manipulated.
It is likely that the administrator user information for the e-commerce site's database was leaked, or that records were illegally updated through SQL injection.
If it is the former, the situation may be largely uncontrollable at this point.

In parallel, the product pages were hacked with misleading login forms, and the page layouts were changed to eye-straining designs favored by attackers...
and ultimately, the service went down, and the first set ended in tatters. It was a
truly awful experience. It was fun.

The results of the first set were stagnant for all teams.

The defense column increases or decreases depending on how many evaluation items are cleared.
This time, there was no defense, so our team's defense score is 0 points.
However, being too defensive may affect the purchase step of the regular crawler, so it seems that just because you are defensive doesn't necessarily mean that your SLA will improve.

〇 Second and Third Sets
In the second and third sets, we hadn't yet identified the specific intrusion routes or taken any fundamental countermeasures, so our approach was simply, "Let's work together and endure without interrupting the service!" We completed various configuration adjustments and applied security patches during the preparation period.

As mentioned above, attacks and incidents occur at specific times according to a timeline, so once you have taken measures, you will likely feel relatively at ease in the future

However, the service outages continued.
Nevertheless, we did manage to temporarily halt some of the intrusions and tampering.

〇 Sets 4 & 5
Based on the results above, for Sets 4 and 5, we decided to review the areas for improvement and develop further improvement and mitigation measures based on the vulnerabilities and incidents that surfaced up to Set 3.

It's especially critical if the service stops at any point, so this must be avoided.
Since the issue is returning a timeout before reaching the web server, I suspect that some problem might be occurring at a different layer.

Since the competitive environment involves multiple servers, I think it would have been better in hindsight if we had thoroughly understood and organized the specifications within the team...
In the end, the improvements and mitigation measures applied in this set were not very effective, and we could not avoid a decrease in the SLA...

〇 Before the final set of the sixth set,
we had the opportunity to receive a partial hint, and we decided to re-examine the issue based on that hint.
It turned out that simply restarting a certain service resolved the problem.

I deeply regret, even as I write this blog, that I neglected to thoroughly examine the service logs in question because I was preoccupied with dealing with other incidents, especially since it was a service I had targeted...
Afterwards, it seems that the aforementioned measures were well-received, and the SLA improved without stagnation.

The results, including the final set, are as follows.


As is often said, players only perform at 50% of their capacity at the beginning, and we can see that there is more than a twofold difference in scores between the first and sixth sets for each team.
This highlights the importance of repeated practice.

In fact, I think the team was able to respond smoothly to incidents in the first and sixth sets

Tips for enjoying Micro Hardening to the fullest

Below are some tips (including Inusuki's reflections) for those planning to attend future events to help you enjoy Micro Hardening 100%

1. Some prior knowledge is necessary.
If you are not an engineer or a beginner, it would be a good idea to familiarize yourself with the usage of the commands covered in the LPIC-1 exam.

The exercises cover basic specifications and provide additional information on how to restart services, but do not go so far as to explain commands

2. Team building is extremely important.
Let's finalize the purpose of the set, who will play which role, and how we will approach the next set.

In this case, we only created a timeline in the first set, and didn't take the time to understand and organize the specifications within the team. This had a big impact later on, and we realized that this was a problem. Even if each person understands the specifications, it is important to look them over as a team and align your understanding

3. Keep a record of your work.
This can be done using Slack or Google Sheets, but make sure to keep a record.
It's important for organizing who did what later on.

4. Consider using external services and tools.
To identify problems and maintain service quality, it is more practical to make full use of external services and tools.

Don't think you can manage everything on your own like Inusuki did.
The teams that rose to the top of the scoreboard were good at using these kinds of tactics.

5. Focus on meeting business requirements. Simply
having the service running won't improve your score.
Take some time to think about how to improve your score and what it means to meet business requirements.

summary

It may be difficult to immediately implement a competition-level environment, but I would like to at least hold a security incident experience session within the company

Thank you for reading this far!