How we talk about CPU (or processors, or cores, or...)

How we talk about CPU (or processors, or cores, or...)

There are a bunch of ways that we talk about CPU, and they're not all measuring the same thing. We're not even always talking about the same physical part of the computer! More on that in a bit, though. Let's talk about the numbers first.

There's the load average sense, where you get to see the run queue averaged over three spans of time. Those are the number of things which are actually running or are trying to run. This has changed subtly in some places over the years to include or exclude different things.

A load average can be particularly fun since it usually (on Linux) includes stuff that's in the kernel and trying to run but is stuck on some kind of lock, like if reading /proc/pid/cmdline gets stuck, or a NFS mount is hung, or whatever. If you stack a bunch of those up from cron jobs or other repetitive things, you might end up with a load average in the hundreds or thousands... and still have a perfectly responsive machine!

On the other hand, if you have just a couple of jobs, but they are consuming every resource they can, then you might have a load average barely higher than the CPU (or processor, or thread... see below) count on the box and yet it'll take 5 minutes to just display the shell prompt when you log in.

Clearly, the load average is only one way to look at things.

Then there's what you see when you look at a process in 'ps'. It shows a number like "%CPU", but that's not instantaneous. It's usually the result of dividing the elapsed CPU time into the elapsed wall time. If your process has been running for a day and has used 1000 seconds of CPU time, then that's about 1% CPU utilization over that whole period.

If the process suddenly ramps up and gets very busy, that number will go up, but if it's been running for a long time, it won't happen right away. After all, when the denominator is large, incrementing the numerator doesn't always make the outcome change a lot.

The numbers from 'ps' are also really fun because a process with a bunch of threads can totally use more than 100% CPU over time. If you have 32 CPUs (or processors, or whatever, read on for more on that vocabulary issue) on your machine, you could totally have a single process approaching "3200% CPU" in a ps output. (Ah, my old web server machines.)

Meanwhile, a tool like 'top' (or atop) is going to show you what's going on more-or-less right now. I suspect it's taking snapshots, calculates the delta in CPU time and then divides it into the amount of elapsed wall time between those snapshots. If the program used 1 second of CPU time in the past 2 real (wall time) seconds, then that's 50%. You can tell that top is doing this kind of shorter-timescale analysis by starting something that chews CPU, watching it spike, and then suspend it and watch it drop back to 0. 'ps', meanwhile, would still be showing the amount used over the entire lifetime of the process so far.

Next, you have process scheduling things like Borg back in the old days (like, over a decade ago). This is where we'd define a job and say "cpu=1.0", and that basically meant it reserved approximately one core for itself. It wasn't an exact science. Sometimes people would do fun things like "cpu=0.99" so they'd be able to use almost the same amount of CPU time but it wouldn't need to reserve a whole core for it.

We also have tools like WMCPULoad, which is a fun little X widget from the days of Window Maker. Oh yeah, I'm talking about the late '90s here. It just shows a single % number which ranges from 0 to 100. That number is talking about the entire box, no matter how many CPUs you have!

I have an eight-way machine that used to do a lot of software-defined radio work. At its busiest, wmcpuload used to show 50% or more. That basically meant that four of its processors were cranking away full-tilt, or equivalent. In all likelihood, all eight of them were doing stuff, and some of them might have been fully pegged, but the others were somewhere less, and it all summed up to 50% of its absolute peak.

So now let's talk about these hardware names: packages, CPUs, processors, cores, and threads. These things show up all over the place.

If you look in /proc/cpuinfo on a Linux box, you will see stanzas labeled "processor", but it's not quite that simple, because a CPU with hyperthreading actually shows up with multiple "processors". My eight-way box is actually a "four core" machine with two "threads" per. These are not the same threads as the ones that we talk about when running processes in parallel, but their numbers definitely influence each other -- if you have a lot of (process) threads, a machine with more (CPU) threads will probably be able to make more progress in parallel than one with fewer. Confusing, right?

Try explaining this to someone who's never heard any of this before.

The "package", then, is the physical thing you slap into a socket or slot on the motherboard (or not, if it's soldered down). Your machine might have multiple packages. Each package might have multiple cores. Then each core might have multiple threads.

What you see in /proc/cpuinfo on Linux... is that last number, all the way at the bottom of the multiplication.

All of this means that what someone says "this thing is at half a CPU", it probably means that it's consuming 50% of one thread, while saying "this thing is using half the box", it's probably using 50% of *everything*.

Then, if someone talks about CPUs, or processors, or threads, or packages, you really have to pay attention because they might be mixing and matching things, while someone else has a slightly different take on this whole business while using the same words.

So, top and atop show you almost current utilization (unless you're back in atop's history), ps shows you total utilization for the entire lifetime, load averages talk about what's running and also *trying* to run, even if it's not doing any work, and resource schedulers sometimes have weird inflection points at integer boundaries. What a mess!

Don't even get me started on virtualization and stolen time. Ugh!