When I first got my hands on Visual Studio 2012, I was trying out every feature I could, new and old alike. I love the new Test Explorer. Code coverage is SO much better than it was before. Code analysis for code clones is a terrific, too. The only thing I wasn’t happy about was performance analysis.
The reason I wasn’t happy with performance analysis is that I couldn’t use it on unit tests. Luckily, Microsoft fixed that with the release of Visual Studio 2012 Update 1. Now you can right-click a test and choose Profile Test to run performance analysis on a single unit test, and it is awesome!
When you choose to profile a test, the test runs as usual followed by the analysis. When analysis completes, you’re presented with a summary view that shows you CPU usage over time, the “hot path” call tree—a tree of the most active function calls where most of the work was performed—and a list of functions responsible for doing the most individual work.
You can find out more and investigate resource spikes by selecting a time range on the graph and filtering. That’s all good and well, but what really blew me away was that you can click on the functions with the most individual work to drill into them. Drill into them? Yea—you’re taken to a view that shows you the selected function, its callers, and the methods it calls. There are percentages that show how much time was taken in each of the three areas (callers, current method, and callees), and you can click into any of the methods displayed to navigate up or down the call stack. The actual code for the current method is also displayed. The only thing that seemed sub-optimal is that I couldn’t edit the code directly; there’s a link to the actual file, though, so you’re only a click away from the editable code file.
There are other, sortable views you can look at, too. You can view a call tree or breakdown by module, and you can get to the same function details view described above from each of those views. It’s a really useful, powerful experience.
Here’s where it gets really nuts, though: add SpecFlow to the mix. SpecFlow lets you write feature-based scenarios that are capable of serving as automated integration tests. The scenarios run like normal unit tests. You can right-click them in the Test Explorer to run performance analysis on them. This means that you can do targeted performance analysis on specific features of your application! To test this out, I sorted my unit tests by execution duration and analyzed the slowest. I was able to find a bottleneck with a caching algorithm used by nearly all of the other modules in the application. Execution time of the 350 unit tests in the project went from 50 seconds to 20. That’s a HUGE improvement from fixing one flaw found from running analysis on one function picked only because it was the most time-consuming in the slowest test.
Good tests are supposed to run quickly, and performance analysis is an invaluable tool to help you triage valuable-but-poor-performing tests. Also, since you’ve got automated tests, you can refactor and optimize the performance of your application with high confidence. If you haven’t used performance analysis before—more specifically, performance analysis for unit tests—give it a shot; I’d be blown away if you didn’t find incredible value.
6 thoughts on “Performance Profiling for Unit Tests”
I have a problem with signed assemblies. It looks like the tests are always executed with instrumentation but do not resign the assemblies after instrumentation. Do you know how I can change the profiler settings (the once used by the test explorer)?
Jan — Sorry, I’m not familiar with that problem. I did a quick search, and it looks like there are a few different solutions offered here: http://stackoverflow.com/questions/2711484/vs-2010-profiling-problem-with-signed-assemblies
I am using VS2012 Pro, and I have Update 3. Yet I do not have this option and would like it. Is it only available for specific editions?
Unfortunately, per MSDN, this feature is only available in the Premium and Ultimate editions.
How can I change the profile settings to get memory allocation information as part of this option? I can change the settings when setting up a custom profiling session, but the “Profile Test” option seems to use some default settings that are hopefully stored somewhere.