Understanding CPU Steal Time - when should you be worried?
July 25 By Derek Posted in HowTo Comments
A big thanks to Eric Lindvall of Papertrail for adding steal time to Scout's CPU Usage Plugin and helping out on this blog post!
Netflix tracks CPU Steal Time closely. In fact, if steal time exceeds their chosen threshold, they shut down the virtual machine and restart on a different physical server.
If you deploy to a virtualized environment (for example, Amazon EC2), steal time is a metric you'll want to watch. If this number is high, performance can suffer significantly. What is steal time? What causes high steal time? When should you be worried (and what should you do)?
Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.
Your virtual machine (VM) shares resources with other instances on a single host in a virtualized environment. One of the resources it shares is CPU Cycles. If your VM is one of four equally sized VMs on a physical server, its CPU usage isn't capped at 25% of all CPU cycles - it can be allowed to use more than its proportion of CPU cycles (versus memory usage, which does have hard limits).
When you run the Linux top command, you'll see a realtime view of key performance metrics. One of the lines is for the CPU:
Two metrics you might have some experience with already are %id (percent idle) and %wa (percent I/O wait). If %id is low, the CPU is working hard and doesn't have much excess capacity. If %wa is high, the CPU is ready to run, but is waiting on I/O access to complete (like fetching rows from a database table stored on the disk).
%st , or percent steal time is the last CPU metric displayed.
You've purchased tickets to the latest Hollywood blockbuster. There are two lines and one ticket booth:
If we applied a CPU steal time-like metric to the ticketing process, it would look like this:
- 0% Steal Time - it's a Wednesday matinee: the ticket booth is picking a moviegoer from line 1, then line 2, then line 1, then line 2, and so on. No one is waiting.
- 50% Steal Time - It's Friday night: instead of being able to purchase a ticket immediately, half of the time a person in the line needs to wait for the person at the booth to complete their purchase. Things are taking longer.
- 100% Steal Time - It's a Friday night and the cash register is broken: no one is moving.
Why is high steal time particularly bad for web apps?
If you have a long-running background computational task that is on an underutilized physical server, it may get access to more than it's share of CPU cycles for a while. Later on, the other VMs need their share of CPU Cycles, so the long-running task will run slower. This might not be a deal-breaker for a long-running task: it might take a bit longer or it might even finish faster (since it was able to use more resources earlier).
However, for web apps, this can bring things to halt. For tasks that need to be performed in real-time, like rapidly serving many web requests, a 4x decrease in performance can cause major backups in request queues, which can lead to outages.
There are two possible causes:
- You need a larger VM with more CPU resources (you are the problem).
- The physical server is over-sold and the virtual machines are aggressively competing for resources (you are not the problem).
The catch: you can't tell which case your situation falls under by just watching the impacted instance's CPU metrics. This is easiest to tell when you have multiple, identical servers performing the same roles, each residing on a different host:
- Has %st (CPU Steal Time Percentage) increased on every virtual server? This means your virtual machines are using more CPU. You need to increase the CPU resources for your VMs.
- Has %st (CPU Steal Time Percentage) increased dramatically on only a subset of servers? This means the physical servers may be oversold. Move the VM to another physical server.
A general rule of thumb - if steal time is greater than 10% for 20 minutes, the VM is likely in a state that it is running slower than it should.
When this happens:
- Shut down the instance and move it to another physical server
- If steal time remains high, increase the CPU resources
- If steal time remains high, contact your hosting provider. Your host may be overselling physical servers.
Scout's CPU Usage Plugin reports key CPU metrics, including steal time. You can create a trigger to alert you of spikes in steal time.
In a virtual environment, CPU cycles are shared across virtual machines on the server. If your VM displays a high %st in top (steal time), this means CPU cycles are being taken away from your VM to serve other purposes. You may be using more than your share of CPU resources or the physical server may be over-sold. Move the VM to another physical server. If steal time remains high, try giving the VM more CPU resources.
Posted by Bernie on May 6, 2008 04:46 PM |
A lthough I enjoyed Failure to Launch (2006), I disagreed with the premise of the movie that there is something wrong with a thirty-something child who does not want to leave home. Children Should live with parents until they die. That's right; until they (the parents) die.
The trouble with American kids and here I speak of 20 and 30 year olds, is that the moment they graduate high school, get a job, or meet their 'true love,' they want to flee the nest. On this forum the question was asked, "What is the average age to move out?" and 66% responded with 20 years old or less with many posters suggesting that even 17 or 18 is a good age to leave home.
In Italy, eight out of 10 Italians aged under 30 still live at home, and the average age for moving out is 36:
"Let's get these big babies out of the house," said Mr Padoa-Schioppa, who left home himself at the age of 19 to work in Germany.
"If young people stay with their parents, they do not get married or have any independence," he added.
Italian men make up the bulk of those staying at home, at around 67 per cent, and a mocking phrase has even been coined to describe them: "Mammoni" or "Big Mummy's boys".
Next year's budget will offer almost ВЈ700 [
$1400] in tax relief to Italians under 30 earning less than ВЈ10,500 [
$21,000] a year, and half that to those earning more.
In addition, the government will pay 19 per cent towards the cost of renting accommodation for university students if they study at least 65 miles away from their home.
Here is how it looks in Europe:
Percentage of 18-24 year olds still living at home
- Italy 95%
I have two sons, two daughters-in-law, and two grandchildren living with my wife and me. My younger son had left a few years ago after he got married but after blowing $55,000 on rent, utilities, and food in two years he realized that it would take many years before he could save up to buy a decent home so he moved back in last year.
Three families can live cheaper than one. Think about it. Taxes on one large lot with a three-family home are cheaper than for three separate one-family homes. Less food is wasted, certainly nothing is left long enough in the fridge to go bad. I pay the mortgage so my boys can save for their future. My housing costs are a lot less than three separate bills would have been for rent and utilities.
We all have cellphones but if we lived apart we each would have needed one land-line for emergencies or for billing purposes for DirecTV. Now we only need one. As for satellite TV, each additional DirecTV connection only costs $5.00. We have 8 DTRs and subscribe to the highest package you can possibly get and it still costs less than if we had 3 separate DirecTV accounts. Each grandchild has her own DTR or recorder so they can tape (that's an old word, no?) their favorite Hannah Montana shows without bothering mom or dad.
From May to September all of us enjoy a 28 foot in-ground pool in the backyard. It costs me the same amount of money to fill and heat the water whether I live alone or with my boys.
We have a 10 TB media server with 1500 movies, thousands of recorded TV shows, videos, photos, etc. that would have been three times more expensive if we had to triplicate it. When we buy a Blu-ray disc we save a copy to the media server and store the disc for safekeeping. We share the cost of buying among us. Even when we rent a movie, there is anywhere from 1 to 3 families watching it.
None of my boys have any debt. With 6 adults we get by quite easily with 4 cars. Normally each family would have needed 2 cars for work, shopping, etc. Five of us work, but with car-pooling we get by with 3 cars and always have one extra as a spare. If we all want to eat out at the same place out of town, 2 cars are enough for the eight of us. Had we lived separately in different cities we would have met up at the appointed restaurant with three cars.
We could get by with one washer and dryer but we have 2 of each just in case. Again, if we lived separately there would have been 3 sets of washer/dryers.
We only need one vacuum cleaner, one set of tools such as a hammer, screwdriver, drill set. The three families share two refrigerators and when Costco has a sale on 5 pounds of grapes I don't have to worry that they'll spoil before they can all be eaten.
If my son and daughter-in-law want to go see a movie or dine out alone, there is no need to find a baby-sitter, grandma and pop-pop are there. Arrangements can always be made. If the grandparents are retired there is no need for daycare.
The immigrant families that live in my neighborhood are a lot smarter than the average native-born American: they tend to live with three generations in a household, grandparents, parents and grandchildren. They save their money, work together in a business, pool their energies, husband their resources, and succeed in America better than the average non-immigrant.
I know that for many young Americans there is a stigma attached to being a momma's boy, but my boys laugh at that notion all the way to the bank. I cannot tell you how many times whenever we have a party over at the house that my sons' friends come up to me and ask if I can "adopt" them.
It is not just my sons who enjoy more of the finer things in life, the fact is it applies to all those who live with their parents; let me give you for example, the study of college kids who live with their parents [Harris Interactive]:
In every instance, college kids who live with their parents have more TVs, computers, DVD players, games consoles and so on than their fellow students. In addition, students who stay at home spend thousands less on non-educational expenses [Statistics Canada]: "Not surprisingly, living with parents or guardians while attending school had a big impact on non-educational expenditures at all levels of education. The average full-time university student living with their parents spent $4,400 on non-educational items compared to just over $8,160 for those who did not live with their parents. Median costs for full-time college students living at home were $3,700 compared to $8,100 for those living away from home."
As for Europe, although young people there generally stay home more than their counterparts here in America, economic troubles in the past few years have caused more of them to say home even longer:
BBC News, 12 Mar 2003, Young 'cannot afford to leave home'
The rising costs of housing and further education together with a reduction in state support is making it increasingly hard for people in the 18-24 age group to strike out on their own.
The research, by independent market analyst Datamonitor, found that 67% of 18-24 year olds across Europe still relied on their parents for housing last year.
There is also an increasing likelihood that young people in Northern Europe return home having left, a phenomenon described as 'boomeranging'.
Here's a snapshot for the US of 1990: Age is the strongest predictor of leaving home. According to the U.S. Bureau of Census, 74% of 18-19 year olds in 1990 lived with parents, compared to 40% at ages 20-24 and 16% at ages 25-29.
By 2000, the figures barely changed. If there is a perception that more young adults live with their parents now than ever before, it is inaccurate, for more detail read Po Bronson, Failure to Launch -- or Failure to Respect the Facts?
Another predictor other than age in the US of how long an adult child will live with his parents is geographic; of U.S. 18-34-Year-Olds Living with Parents (2006): 35% live in the Northeast, 27% in the Midwest, 25.5% in the South, and 24.5% in the West.
I know I have not mentioned the cons, some parents can find the dress and living habits of their children intolerable. Adult children may bristle under some fuddy-duddy "rules" their parents may want them to adhere to. But if parents want to avoid the empty-nest syndrome and want their married children to stay home they must make accommodations; parents cannot treat their married children as if they were infants. The new son- or daughter-in-law will have a greater influence on their children than the parents used to and this must be accepted.
Living with anyone is difficult. The high divorce rates and the number of children living in broken homes is staggering. This is not the way for a society to flourish. There is obviously something wrong. Living in a larger family setting may help with finances and offer more free time for the just-starting husband and wife. Certainly issues of money, careers, and sexual problems contribute to divorce. Living in a big home, women can more easily raise a child and still pursue a career. Having grandparents to help with the kids can give parents more free time for themselves.
Grandparents, instead of having useless lives with nothing to do can now have fun with their grand-kids instead of seeing them on two or three holidays a year. Once or twice a week my granddaughter comes to our bedroom to sleep with us. We watch her favorite Disney films until she falls asleep. The time this gives her parents to be alone is precious beyond measurement. Perhaps it's not so surprising that parents burn out, what with each having a job and trying to take care of home and child and no free time to snuggle and be with each other alone.
Grandparents can enjoy shopping again. Last year I saw an old woman buying a small package of Corn Flakes and noted that the larger package would give her twice as much cereal for only a few pennies more. When I suggested this to her, she told me that it's wasteful to buy the larger version since she could hardly finish the size box she was buying before it goes stale.
You can tell who lives alone by observing the checkout at your supermarket. The single pork chop, the smallest container of milk, the half-loaf of bread. An entire life tallied on a shopping receipt.
Both of my sons are the most fierce, independent-minded individuals you will ever meet. But living with their parents does not mean they are dependent, helpless, vulnerable human beings. Some of the most fierce, independent-minded individuals who ever existed lived with their sons, daughters, nieces, nephews, cousins, uncles, aunts, parents, altogether in caves or tents or treetops. They shared resources, pooled their hunting, took mutual care of their young. That group grew and prospered until they took dominion over the Earth.
We've come a long way as a species, but in some regard we have lost family. Perhaps one day it will be a stigma to live away from family.
Causes of Divorce:
Time, sex, money biggest obstacles for young married Couples
A recent study by the Creighton University Center for Marriage and Family suggests that time, sex and money pose the three biggest obstacles to satisfaction in the lives of newly married couples. The study found that debt brought into marriage, the couples' financial situation, balancing job and family, and frequency of sexual relations were of greatest concern to those ages 29 and under. Those age 30 and over shared with their younger cohorts the concerns of balancing job and family and frequency of sexual relations, but also added as problem areas constant bickering and expectations about household tasks.
New grads find home is sweet, Excerpt: "63 percent of U.S. college students said they planned to live at home, according to an online poll by MonsterTRAK.com, a job search company."
Living with your grown children can also help keep grandparents married to each other:
The sad news is that there has been a steady increase in the number of divorces among couples married 30 or more years. Many long-term married couples divorce one another after the kids leave home.
They realize too late that their children kept them together. Other couples divorce during the empty nest years because they can't handle the health issues and the sense of an uncertain future along with being overwhelmed by too much togetherness.