[EC2] What is the CPU usage steal item?

This is Nakagawa from the System Solutions Department.

Previously, there was a frequent CPU load on a certain server, so I investigated and
found that it was due to the characteristics of T2 instances of AWS's EC2.
Since this was a good opportunity, I decided to investigate the causes and characteristics of the load.

Result of top command

top - 01:41:32 up 13 min, 1 user, load average: 0.17, 0.17, 0.08
Tasks: 88 total, 1 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 2.0%sy , 0.0%ni, 93.8%id, 3.0%wa, 0.0%hi, 0.0%si, 0.5%st
Mem: 1017372k total, 184052k used, 833320k free, 33848k buffers
Swap: 1048572k total, 52416k used, 996156k free, 688k cached

It seems that the item "st(steal)" at the right end of the Cpu(s): line is related.

CPU steal

I wasn't familiar with this item, so I looked into the details, and it

turned out that in a virtual server, it means
"the percentage of time that CPU resources were not allocated even when requests were made to the guest OS for processing EC2 has various instance types, and among them, T2 instances
have the ability to operate with limited performance (burst) when usage exceeds a certain level.

CPU credits

As I investigated the T2-based instances, I again encountered an unfamiliar word.
According to the official website represents the performance of an entire CPU core for one minute ."

It bursts when all the allocated CPU credits are consumed, so
I roughly think of the numbers as a guide to CPU usage.
When I think of a CPU, I thought it meant that a certain amount of processing could be secured in proportion to the number of cores, but
that may not be the case with T2-based instances.

How to deal with it

Now that we know the meaning of burst and its activation conditions, how should we respond?
If you do not change the amount processed, there are three possibilities:

・Restart the instance

Rebooting will reallocate CPU credits.
This is only a temporary response, so if left as is, it will eventually burst.

・Instance spec upgrade

Increase the T2 instance size to increase the amount of CPU credits allocated.

・Change instance type

T2 instances use the CPU credit method, and since they cannot process as many cores as they do
, you may want to change to a fixed performance (M3, C3, etc.) to ensure stable processing.

lastly

EC2 is often used as a test environment because everything from instance creation to startup is smooth.

Although we are using it,
we only pay attention to the instance type when it is specified at the time of construction, or when investigating the difference in CPU, memory, and usage charges of other instances.

In writing this blog post, I learned a lot of new things about performance, so
I would like to use this information to improve my skills and knowledge for troubleshooting and work.