FreeBSD Flame Graphs
source link: https://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
FreeBSD Flame Graphs
10 Mar 2015
At the last FreeBSD Developer and Vendor Summit, I gave a talk on "Flame Graphs for FreeBSD", where I summarized the different types (CPU, memory, disk I/O, off-CPU, chain graphs), showed how they can be generated on FreeBSD, and did some live demos. I think it's one of my best talks so far, whether you care about FreeBSD or not, to see how this visualization can be used to navigate different types of profiling data.
The slides are on slideshare:
There was no camera in the room, so I captured the talk from my laptop using screenflow youtube:
(This is the first time I've published a screenflow. What do you think? This might be better than nothing for my demos that aren't otherwise recorded. Thanks to Deirdré Straughan for help editing this.)
We're using flame graphs on the Netflix Open Connect Appliances (OCAs) to understand CPU usage and look for optimizations. Not just CPU flame graphs, but also CPI flame graphs, where the color indicates cycles-per-instruction (CPI). This shows what the CPUs are really doing: are they busy retiring instructions (executing code), or are they stalled on memory I/O?
Flame graphs are easy to generate given the output of a profiler (DTrace or pmcstat), and I included the commands in the talk. For example, to generate a kernel CPU flame graph using DTrace, by sampling kernel stacks at 197 Hertz for 60 seconds:
# git clone https://github.com/brendangregg/FlameGraph # or download zip
# cd FlameGraph
# kldload dtraceall # if needed
# dtrace -x stackframes=100 -n 'profile-197 /arg0/ {
@[stack()] = count(); } tick-60s { exit(0); }' -o out.stacks
# ./stackcollapse.pl out.stacks | ./flamegraph.pl > out.svg
Here's an example resulting flame graph (SVG). Click to zoom:
And to generate a stall cycle flame graph using pmcstat, by sampling stacks based on resource stalls:
… # pmcstat –S RESOURCE_STALLS.ANY -O out.pmcstat sleep 10 # pmcstat -R out.pmcstat -z100 -G out.stacks # ./stackcollapse-pmc.pl out.stacks | ./flamegraph.pl > out.svg
Many other types are possible, as I covered in the talk.
We've automated CPU flame graph generation on the OCAs (thanks, Scott), so that they can not only be created easily, but also used for non-regression analysis.
I had done an epic flame graphs talk at USENIX/LISA 2013 (slides, video) lasting 90 minutes. It was really two talks in one: CPU flame graphs, then other flame graph types. I was thorough and explained everything, but I've since hesitated to encourage people with casual interest to watch a 90 minute video. For the FreeBSD dev summit, I gave a tour of all flame graph types in 50 minutes, and really got stuck into the technical details fast, skipping introductions. It was a lot of fun, and perhaps I should be doing more talks like this.
This FreeBSD Developer and Vendor Summit talk was a day after my performance analysis talk at MeetBSD CA, which I also recommend watching.
In a follow up post, I'll discuss the new method for creating off-CPU flame graphs using procstat that I demoed during the talk.
Recommend
-
11
Coloring Flame Graphs: Code Hues 30 Jul 2017 I recently improved flame graph code coloring. If you're automating or implementing flame graphs, this is a small detail that may interest you. (For an intro to flame graphs, see...
-
10
USENIX/LISA 2013 Blazing Performance with Flame Graphs 23 Apr 2017 In 2013 I gave a plenary at USENIX LISA on flame graphs: my visualiza...
-
9
Yesterday I posted about flame graphs for file systems, showing how they can visualize where disk space is consumed. Many people have
-
5
Where has my disk space gone? Flame graphs for file systems 05 Feb 2017 My laptop was recently running low on available disk space, and it was a mystery as to why. I have different tools to explore the file system, including...
-
7
Unikernel Profiling: Flame Graphs from dom0 27 Jan 2016 Is a unikernel an impenetrable black box, resistant to profilers, tracers, fire, and poison? At SCaLE14x this wee...
-
8
Java Mixed-Mode Flame Graphs at Netflix, JavaOne 2015 06 Nov 2015 At JavaOne this year I gave a talk on Java mixed-mode flame graphs, which we are rolling out at Netflix for CPU analysis and more. These make use of a new fea...
-
8
FreeBSD Off-CPU Flame Graphs 12 Mar 2015 While I was giving a talk on FreeBSD flame graphs, a new technique was suggested by the audience for gathering off-CPU stacks, which is something I'd like to see improved and included...
-
6
CPI Flame Graphs: Catching Your CPUs Napping 31 Oct 2014 When you see high CPU usage, you may be forgiven for believing that your CPUs must be furiously executing software, like obedient robots that never rest. In reality, y...
-
10
node.js Flame Graphs on Linux 17 Sep 2014 CPU flame graphs are a useful visualization application stack traces, allowing you to quickly identify and quantify what to...
-
3
0:01 / 53:37 ...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK