If you’ve ever used any profiling tools but have never used Shark on OSX, find a Mac and play with it for a while.  In fact, if you can’t find a mac, go to an Apple Store and find a Mac with the SDK installed and play with Shark.  I’ll warn you though; it’ll spoil you.  You will have a hard time using any other profiling tools.

While much of what we do at Mozilla can be profiled using Shark, and have the results be meaningful cross-platform, the graphics subsystem is not one of these.  Under OS X, all of our drawing operations go through Quartz; we don’t go through much of Cairo’s low-level software rasterizer, like we do on Win32.  So, I often have to profile on Win32 when chasing down performance issues.

The best tool available for profiling on Windows today is Intel’s VTune, at least that I can find.  (If there are better tools, please let me know.)  Much like Shark and other similar tools, it collects data whenever a sampling interval expires, either time-based or program counter based.  Crucially, unlike Shark, it does not collect the entire call stack at each sample point, which become an issue later.

The main difference between the two is the UI.  Here’s what Shark looks like when you start it up:

Note that this is with the mini-config editor shown; normally you’d only see the top line.  A Start button, a (sane) default sampling type, whether you want to sample a process or everything, and a list of running processes.  If I wanted to, I could have Shark launch a process for me, which would be useful if I was profiling startup, but there’s no need to bother in this case.

Let’s hit Start and sample Firefox; note that I’m using a debug build with symbols because it’s what I have handy, and am running a simple benchmark.  I fire up the benchmark and hit Option-Esc at the start, and Option-Esc at the end.  Shark will chug a bit, looking up symbols and so forth, and this view will come up:

With no configuration, I have an incredibly useful view of what’s going on in my program.  From this, I can clearly see which functions are taking the most time.  But, because of the call stack sampling I mentioned above, I can drill down and figure out exactly who’s calling these top functions:

A top-down view provides an inverse view of my sampling run.  95% of all samples were in firefox-bin, and they were all from the start() symbol.. and I can quickly see the hot path all the way from the start of my program here:

These views are the simplest ones that Shark provides, but they provide a wealth of information.  Based on this, I have a clear picture of where time is being spent, and can identify potential candidates for optimization.

However, going back to the first view above, the top function is in CoreGraphics; it was called by functions in libRIP.A.dylib.  As those are core OS X system libraries, it’s interesting to know that that’s where my time is spent, but I really care about where the time is spent in parts of the application that I have control over.  Shark has a feature called Data Mining that lets you do exactly this.  For any symbol or library, you can perform a simple operation — charge its time to its callers.  Here’s what the right-click menu looks like for the top symbol:

Note that you can also do things like remove certain call paths from the sampling display, to drill down into cases that you care about that you may not have been able to create direct benchmarks for.  I’m going to go through and charge the system libraries all to callers — CoreGraphics, libRIP, libSystem, mach_kernel, AppKit, libobjc, etc., and I end up with this view:

If the first view, with no work and configration was merely “incredibly useful”, this is mind-bogglingly amazing.  I am looking at where time is being spent in my app, and that’s it.  I can still drill down and see who called _cairo_quartz_surface_fill:

That’s only a small fraction of what Shark can do, but even without knowing any other Shark abilities, you get crucial and useful information for your app.

Let’s run through the same thing with VTune.  Here’s the first screen you see when you launch VTune:

Let’s do some Quick Performance Analysis, and I get sent over to this screen:

My options are to profile the entire system or to select an application to launch.  I don’t want to launch an application (though much of the remainder applies regardless), so let’s profile the entire system.  Firefox is already running, so that should get caught.  I can’t actually even choose call graph data here, because that requires an app to launch; if you do choose both call graph and sampling data, it will run your app twice, the first time to sample, and the second to collect call graph info (with no merging of the data, just two different datasets).  Let’s choose sampling and go:

We’re presented with this UI information overload disaster.  I’m going to make the window quite large to see the data from now on, but this is what we have.  We see that most of the instructions retired samples are in firefox.exe; drilling down into that, 99.9% is inside a single thread, also good.  The problems start on the next view, once I drill down into that thread:

I am presented with a view of all the modules that were included in a sample of this thread.  Why do I care about the module?  I might care to see what module a symbol is in, but I really want all the symbols, not all the modules.  Most of the clockticks are in ntkrnlpa.exe, and the second highest (and most of the instructions) are in igxpdv32.DLL.  I don’t control either of those; the first is a windows kernel module, and the second is (I think) the Intel video driver.  I could obtain symbols for the first from Microsoft, but I have no idea what’s happening in the second.  xul.dll is next, which we’ll jump into.  Note that this was even more useless before libXUL was created; when we had a dozen different component dlls, it was impossible to get any kind of information here.  Even now, some pieces, such as the JS library and NSPR, are separate.  Let’s drill down into xul.dll:

This view is showing me functions that I was executing when a sample was taken for functions that are inside xul.dll.  I have no idea how this relates to the rest of the system; indeed, if any of these functions happened to call a function that took up most of the time inside ntkrnlpa.exe, I’d have no idea which one it would be — the time spent inside ntkrnlpa.exe isn’t represented here.  This data is at best not useful, and at worst damagingly misleading. For example, in the image above, the OnPaint method is showing up as taking almost no time.  In reality, all of the time is inside OnPaint’s descendants, but VTune doesn’t have this information.  Doing a Call Graph profiling run reveals some of this, but it suffers from many similar UI difficulties, including being able to see only immediate ancestors and immediate children of a given function.

It’s entirely possible that my difficulties with VTune are because I just don’t know how to use it.  I’d be happy to admit this, if someone could show me how to get the same data that I can get out of Shark out of VTune.  But as it is, VTune is incredibly frustrating to use for optimizing any kind of application.  I can see it being useful for doing microoptimization on specific inner loops that can be isolated into a stand-alone benchmark, but Shark handles that case fine as well.

On top of all of this, Shark comes with the free OS X SDK, whereas VTune is a $700 product with an annual $280 support renewal (required to receive upgrades, even for the major version you purchased!).  I hope that the VTune folks bite the bullet and just copy Shark’s UI for the next version; why mess with something that works really well?  There’s still plenty of innovation that would be possible, especially in the area of optimization assistants and the like, but for pure profiling work, give me Shark.