[Micro Hardening v2] Gentle security incident experience

Hello, this is Inusuki from the System Solutions Department.

Security incidents are something that is familiar to users; one day, the service they are using becomes the target of a cyberattack, making the service unavailable or causing some functions to be stopped and operated in a degraded manner. That's what happens.

This time, I participated in the event "Micro Hardening" where you can actually experience such a security incident, so I summarized my experience in a blog.

Refusal

・In order to avoid spoilers, this blog does not include actual examples of attacks carried out with Micro Hardening or detailed countermeasures.
・Contains much of Inusuki's personal opinion.
・There are 6 sets of exercises, but we will send them to you as a digest.

What is Micro Hardening?

One of the Hardening sub-projects, this is an exercise in which teams of 1 to 4 people protect a small environment configured on the cloud from various cyber attacks and hone their ability to respond in the event of an attack. The same attack is performed each time over a short period of time and in multiple sets.

The original Hardening Project has a high level of required level and participation hurdles, so I think Micro Hardening is a hardening event that is recommended for non-engineers and people who are just starting to learn about security, as it allows you to easily participate even if you have no incident response experience. Masu.

Differences by version

Micro Hardening v2 has more incidents than v1.

SLA (Service Level Agreement) has been newly introduced as a standard to meet business requirements In competitions, the emphasis was more on meeting set business requirements (operating hours and sales) than on service quality.

Main story

introduction

In order to head to Awaji Island, where the competition venue is located, we transferred to the subway, JR, and bus to arrive there.
the day beforefrightened by numerous incidentsI was having so much fun that I couldn't sleep, so I took a nap on the bus after the transfer.

That aside, Awaji Island.
As it is a popular place for tourists, the location was very good, and the competition venue, SAKIA, could overlook the Seto Inland Sea.

In addition, SAKIA is a community facility that was renovated from a closed elementary school and includes a restaurant, hotel, and co-working space, giving it a friendly atmosphere.

Preliminary explanation and team division

There was an explanation of the event and team announcements based on advance materials.
(The main part of the document is about competition environment information, so it will be omitted here.)

The team composition was announced on the day, so we couldn't communicate in advance, but Team 1, where Inusuki was, had three programmers and Inusuki.

It was interesting to talk to them because everyone had different backgrounds and reasons for participating in the event.

Event starts

〇 1st set

"GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__aaaaaaa___________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a____a_a____aa___aa__a_________________ HTTP/1.1" 40 4 486 "-" "curl/7.29.0" "GET /____a_________a___a___a_a_a_a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a__aaaa__a_____a__a__a__a__aaaaa_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0 " "GET /____a_____a__aaaaaa__a_____a__a_________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a__a_____a__a_____a__a___ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa___a_____a__a_____a__aaaa aaa___________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa___aaaaaa_____a_____aaaaaa___aaaaaaa__ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a_____a_______a_a____a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_ __________a______a___a___a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__aaaaaa______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /__________a_____a_____aaaaaaa__a___a_______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /____a_____a_____a_____a_____a__a____a______a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_____aaaaa______a_____a_____a__a_____a_____a_____ HTTP/1.1" 404 486 "-" "curl/7.29.0" "GET /_________________________________________________ HTTP/1.1" 404 486 "-" "curl/7.29.0"

The ASCII art of [GAME START], which is output to the access log when the game starts, has started...Micro Hardening.
For the time being, I decided to proceed with no guard at first, based on the policy of ``watching'', organizing the events, and creating a timeline.

A few minutes have passed since the start.
No particular incidents occurred, and I felt less nervous inside.
(Inusuki, his girlfriend at the time, didn't know that this was the calm before the storm.)

A short while later, we received a report that the site title had been altered within the team...
When we reloaded the e-commerce site, the words ``hacked anonymous'' were displayed in the site title.
What a bold declaration of war.

When we investigated access logs from around the relevant period, we found a request to a suspicious URI.
Moreover, it returns an HTTP status code of 200.
When I looked under the document root, there was a php file with a common file name that I didn't recognize.
It appears that some kind of vulnerability was exploited to set up a backdoor.

Meanwhile, an incident occurred in which the price of a product on an e-commerce site was fraudulently manipulated.
It is likely that the administrative user information for the e-commerce site database was leaked, or that records were updated incorrectly by SQL injection.
If it's the former, you may not be able to collect much at this point.

At the same time, a login form that misleads users is embedded in the product page, or the page layout is changed to the attacker's favorite one that makes the eyes flicker...
and eventually the service goes down. And so, I finished the first set with my hands in tatters.
I had a really bad experience. fun.

The results of the first set were stagnant for all teams.

The defense column increases or decreases depending on how many evaluation items you clear.
Since there was no guard this time, my team's defense points were 0 points.
However, if you seriously defend it, it may affect the purchase step of the official crawler, and it does not seem that the SLA will necessarily improve even if you defend it.

〇 2nd and 3rd SetsIn
the 2nd and 3rd sets, we had no idea how to identify specific intrusion routes or take fundamental countermeasures, so our policy was to ``divide our efforts and endure without suspending services.'' ! ” During the preparation period, I completed various settings corrections and applied security patches.

As mentioned above, specific attacks and incidents occur at specific timings based on the timeline, so once countermeasures are taken, I think things will be relatively easy later on.

......However, the issue of service outage remained the same.
However, although provisionally, we succeeded in stopping some of the intrusions and tampering.

〇 4th and 5th Sets
Based on the above results, we decided to review the areas for improvement in the 4th and 5th sets, and devise improvement/mitigation measures again based on the vulnerabilities and incidents that had surfaced by the 3rd set. Ta.

In particular, it is fatal for the service to stop at a certain point, so this must be avoided.
Since a timeout was returned before the event reached the web server, I thought that some kind of problem might be occurring in a different layer.

Since multiple servers coexist in the competitive environment, I think it would have been better if the team had understood and organized the specifications in detail within the team...
In the end, the improvements applied in this set... - Mitigation measures were not very effective, and the SLA could not be avoided...

〇 I had a chance to receive a partial hint before the final set of the 6th set
, and based on that hint I decided to check again.
Judging from the results, the problem was resolved simply by restarting a certain service.

Even as I am writing this blog, I deeply regret that I neglected to carefully examine the logs of the service in question because I was focused on responding to other incidents. It seems that the measures were well received, and the SLA improved without any stagnation.

The results including the final set are as follows.


As it is said that at first a team can only perform at 50%, you can see that each team's score is more than double the difference between the 1st and 6th sets.
This shows the importance of repeated efforts.

In fact, I think the team was able to respond smoothly to incidents in the first and sixth sets.

Tips to fully enjoy Micro Hardening

For those who will be participating in the event in the future, we have summarized the tips (including Inusuki's reflections) below to fully enjoy Micro Hardening.

1. A certain amount of prior knowledge is required
If you are a non-engineer or a beginner, I think it would be better to limit the usage of the commands that will be asked on the LPIC-1.

This is because the exercises cover basic specifications and provide supplementary instructions on how to restart services, but do not include explanations of commands.

2. Team building is really important
. Let's work out the purpose of the set, who will play what role, and how to tackle it in the next set.

In this case, I am keenly aware that the fact that I only created the timeline in the first set and did not take the time to understand and organize the specifications within the team separately had repercussions later on. . Even if each person understands it individually, it is important to go through it as a team and agree on the same understanding.

3. Leave a trail of your work
This can be in Slack or Google Sheets, so keep it.
It is also important to sort out who took what action later.

4. Consider using external services and tools
In order to uncover problems and maintain service quality, it is more realistic to make full use of external services and tools.

Please don't think that you can handle things on your own like Inusuki.
The teams with the highest scores were good at using this area.

5. Be conscious of meeting business requirements
Simply having the service running will not improve your score.
Take a moment to think about how you can improve your score and what it means to meet job requirements.

summary

It may be difficult to implement a competition-level environment right away, but I would like to at least hold a security incident trial session within the company.

Thank you for reading this far!