[EC2] What is the steal item in CPU usage?

This is Nakagawa from the System Solutions Department

Previously, we encountered frequent CPU load issues on a certain server, and upon investigation,
we discovered that it was due to the characteristics of T2-series instances within AWS EC2. This presented
a good opportunity to investigate the cause of the load and its characteristics further.

Result of top command

top - 01:41:32 up 13 min, 1 user, load average: 0.17, 0.17, 0.08
Tasks: 88 total, 1 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 2.0%sy, 0.0%ni, 93.8%id, 3.0%wa, 0.0%hi, 0.0%si, 0.5%st
Mem: 1017372k total, 184052k used, 833320k free, 33848k buffers
Swap: 1048572k total, 52416k used, 996156k free, 24688k cached

It seems that the item "st (steal)" on the far right of the Cpu(s): line is related

CPU steal

Since it was an unfamiliar term, I looked into it further and found that in virtual servers,
"the percentage of time when a processing request is made to the guest OS, but CPU resources are not allocated."
it refers to
EC2 has various instance types, and among them, T2 series instances
have a feature that limits performance (bursts) when a certain amount of usage is exceeded.

CPU Credits

While researching T2 instances, I came across yet another unfamiliar term.
the official websiteAccording toto represent "the performance of the entire CPU core over one minuteit seems

as it causes a burst when all allocated CPU credits are consumed
I see this as a rough indicator of CPU usage,
I used to think that CPUs could guarantee a certain amount of processing power proportional to the number of cores, but
that might not be the case with T2 series instances.

How to respond

Now that we understand the meaning and activation conditions of a burst, how do we deal with it?
If we don't change the amount of data being processed, there are three possible approaches:

・Restart the instance

Restarting will reallocate the CPU credits. However, this
is only a temporary solution, and the system will eventually burst if left in this state.

・Upgrading instance specifications

Increase the T2 instance size to increase the amount of CPU credits allocated

・Change the instance type

T2 instances use a CPU credit model, meaning they don't necessarily have the processing power to match the number of cores. Therefore
, you might consider switching to a fixed-performance instance (such as an M3 or C3 instance) to ensure stable processing.

lastly

EC2 is often used as a test environment because the process from creating an instance to launching it is smooth

Although I was using it,
I only paid attention to the instance type when it was specified during setup, or when investigating the CPU, memory count, and differences in usage fees compared to other instances.

While writing this blog post, I learned a lot of new things about performance, which
I hope to use to help with troubleshooting and improve my technical knowledge and skills in my work.