[EC2] What is the CPU usage steal item?

This is Nakagawa from the System Solutions Department.

Previously, a certain server was experiencing frequent CPU loads, so I investigated and
found that it was due to the characteristics of the T2 series EC2 instances on AWS. This
was a good opportunity to investigate the cause and characteristics of the load.

Result of top command

top - 01:41:32 up 13 min, 1 user, load average: 0.17, 0.17, 0.08
Tasks: 88 total, 1 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 2.0%sy, 0.0%ni, 93.8%id, 3.0%wa, 0.0%hi, 0.0%si, 0.5%st
Mem: 1017372k total, 184052k used, 833320k free, 33848k buffers
Swap: 1048572k total, 52416k used, 996156k free, 24688k cached

It seems that the item "st (steal)" on the far right of the Cpu(s): line is related

CPU steal

Since I was unfamiliar with this term, I looked into it in more detail and

found that it means
"the percentage of time that a processing request was made to the guest OS but CPU resources were not allocated" EC2 has a variety of instance types, and among them, the T2 series instances
have a function that limits performance (bursts) when usage exceeds a certain level.

CPU Credits

As I looked into the T2 series instances, I came across another unfamiliar term.
According to the official website represents the performance of one CPU core per minute ."

Since a burst occurs when all allocated CPU credits are consumed,
I think of this as a rough guide to CPU usage.
When it comes to CPU, I thought that a certain amount of processing power could be secured in proportion to the number of cores, but
this may not be the case with T2 series instances.

How to deal with it

Now that we understand the meaning of burst and the conditions under which it occurs, how do we respond?
If we don't change the amount of processing, there are three possible options:

・Restart the instance

Restarting the machine will reallocate CPU credits.
This is a temporary solution, so if you leave it as is, it will eventually burst.

・Upgrading instance specifications

Increase the T2 instance size to increase the amount of CPU credits allocated

・Change the instance type

T2 series instances use a CPU credit system, which means they cannot process the number of cores, so
you may want to consider changing to a fixed performance instance (such as the M3 series or C3 series) to ensure stable processing.

lastly

EC2 is often used as a test environment because the process from creating an instance to launching it is smooth

Although we were using it,
we only paid attention to the instance type when it was specified at the time of construction, or to investigate the difference in usage fees for CPU, memory size, and other instances.

In writing this blog post, I learned a lot of new things about performance, so
I hope to use it to help with troubleshooting and to improve my skills and knowledge in my work.