How to Measure Steal Time for EC2 Oversubscription on Public Cloud
Measure EC2 Steal Time with mpstat to identify oversubscription and noisy neighbors
High Steal Time does not necessarily equal oversubscription; evaluate with Nitro and credit mechanisms.
Measuring Oversubscription with mpstat
To measure Steal Time on a public cloud EC2, the most direct way is to run mpstat -P ALL 1 and watch %steal for a few rounds. But don't shout oversubscription just because the number is high—I've been burned: when a T3 instance exhausts its CPU credits, %steal can spike to 30%+, but that's not neighbor preemption; it's the credit pool running dry. To truly distinguish between normal scheduling overhead under Nitro virtualization, burstable credit exhaustion, and noisy-neighbor preemption, you need to check /proc/schedstat and CloudWatch's CPUCreditBalance metric together.
In practice, I usually start by running sudo apt install sysstat && mpstat 1 5, focusing on whether %steal consistently exceeds 10%. If it's just transient jitter, it's mostly normal CPU resource contention on the host; only when it stays high for a long time and the steal time ratio in cpustat is stable do I suspect oversubscription. Note: High Steal Time does not equal oversubscription—it could also be that you under-provisioned. After enabling Unlimited on a T3 instance, sustained high load will first consume future credits in advance, and the symptom is increased steal.
But that's not the end. I'll also export the mpstat output to CSV and package it with CloudWatch screenshots into a PDF—if you ever need to file a complaint ticket, this is hard evidence. AWS shared instances may look attractive in terms of cost-effectiveness, but when doing FinOps accounting, you need to factor the potential steal risk into a premium rate: with the same 4 vCPU, rather than gambling on getting a quiet neighbor, it's better to calculate the long-term cost difference between Dedicated Host and shared instances.
For the full audit methodology, see the EC2 StealTime Inspection Guide, or go directly to the Operations Audit Console to run an automated round.
Nitro Oversubscription and T3 Credits
The Nitro architecture on AWS EC2 differs from conventional KVM oversubscription: Nitro's CPU scheduling has harder isolation, but neighbor contention still exists on shared hosts. What's really easy to confuse are burstable instances like T3/T3a/T4g. When CPU credits are exhausted and Unlimited is not enabled, performance is forcibly pulled back to the baseline. At this point, %steal in mpstat doesn't necessarily spike; instead, it feels more like your own 'lag.' After enabling Unlimited, credits can go negative, but that incurs additional costs.
So when troubleshooting, first use mpstat 1 for continuous sampling, then cross-check the CloudWatch CPUCreditBalance metric. If steal is high but credit balance is sufficient, that's evidence of oversubscription; otherwise, it may be normal overhead from Nitro virtualization. In practice, if high steal persists, it's recommended to collect /proc/stat timestamp evidence for submitting a support ticket or justifying a downsizing—after all, from a FinOps perspective, paying for oversubscription means the premium rate is not cost-effective.
Forensic Tickets and FinOps Premium
When you catch persistently high steal time on EC2, don't rush to put the "oversold" label on AWS. My habit is: first use mpstat -P ALL 1 for continuous sampling for 15 minutes, then pull CloudWatch's CPUCreditBalance and CPUCreditUsage for comparison. If T3/T4g credit balance is zero, then high steal time is mostly due to CPU credit exhaustion, not neighbor CPU stealing. To truly confirm a "bad neighbor", we need to see under Nitro virtualization whether steal exceeds 5% and is accompanied by irq increase — but Nitro itself has a small scheduling overhead, so don't apply the VPS-style 0.5% threshold rigidly.
At the forensics stage, I would simultaneously export CloudWatch's CPUUtilization and StealTime metrics (EC2's StealTime is a custom namespace, need to use GetMetricData to pull), then use /proc snapshot to record the cpu line. Combine these three by timestamp into a PDF, and attach it directly in the ticket. When AWS Support sees data with a timeline and screenshots, they are usually more willing to investigate the underlying host — although they rarely admit overselling, they will give you an instance change or adjust placement group.
Finally, on the FinOps premium: For a c7i.large of the same spec, running batch processing on a shared host, instances with steal time 3% and 8% can differ in actual throughput by more than 12%. If you factor in the additional billing after credit exhaustion (T-series unlimited), the combined cost may be even more expensive than a dedicated host. It is recommended to include steal time in the monthly cost report, and for those exceeding 5%, convert them as a "performance loss premium", then compare with the annual price of Dedicated Host — many times, upgrading the host for critical tasks is actually more cost-effective.
FAQ
How to measure EC2 Steal Time?
Use the top or vmstat command to view the CPU steal percentage, for example the %st field in top.
Does high Steal Time always mean oversubscription?
Not necessarily. Consider the Nitro architecture and credit mechanisms. High Steal Time may be due to temporary resource contention.
How to determine oversubscription under the Nitro architecture?
Query the instance's underlying physical CPU to vCPU ratio via the AWS API and compare it with the actual allocations.
How does the credit mechanism affect Steal Time?
When T-series burstable instances exhaust their credits, CPU throttling may cause Steal Time to rise; check the credit balance.
What are the steps to comprehensively determine oversubscription?
Measure Steal Time first, then check the physical resources of the Nitro instance, and finally analyze credit usage to draw a comprehensive conclusion.