Performance Profiling Tools, Explained

Attach a sampler to the process that is already slow, wait thirty seconds, and read the widest rectangle in the flame graph. That is the whole of performance profiling, and it is the position this page has to argue against before it can argue for anything else. The case is strong enough to have won by default. Every mainstream runtime now includes a sampling profiler, its overhead is in single digits, and it attaches to a live process without a restart, a rebuild or a compiler flag. If the case were complete, deterministic tracing would be legacy cruft, the flags that switch it on would be archaeology, and reading a flame graph would be the only skill the work requires.

It fails at one point, and it fails where the stakes are highest. A sampling profiler estimates frequency from a fixed number of stack samples, so a function that completes between two intervals can be undercounted or missed outright. The Python 3.15 profiling documentation explains the limit without hedging: statistical profiling may undercount short-lived functions that complete between samples. Remove ten thousand calls from a hot loop and the flame graph can look identical, because the samples never resolved those calls in the first place. Method comes before tool: the question being asked selects the method, and no IDE default can make that choice.

A profile is a measurement, and the instrument changes the reading

A profile is a set of statistics describing how often and for how long parts of a program execute. Two mechanisms produce those statistics, and the difference between them determines what the numbers mean. A sampling profiler interrupts the process on a timer and notes whatever call stack is on the CPU. The other instruments every call, return and exception event and emits a record for each one. The split runs through every performance profiling tool on the market, command line or GUI, however the vendor brands it. The Python 3.15 profiling documentation rates sampling overhead as "virtually none" and deterministic tracing as "moderate", and that gap determines whether the profiler can stay attached to a production process at all.

The same documentation sets the pair against a shared list of attributes, and those values generalise past Python because they follow from the mechanism.

Every row above is taken from the comparison table in the Python 3.15 profiling documentation.

The flat profile ranks routines and ends there.

The profiling chapter from Sun's compiler tools, hosted by the University of Edinburgh's School of Informatics, orders its levels from simplest to most detailed, and the ordering is the useful part. prof reports which routines in a program are most time-intensive. gprof adds a call graph above the same flat profile. tcov reports how often each source statement ran. Profile feedback closes the loop, feeding the runtime record back to the compiler for optimisation. The worked example in that chapter compiles a small index program, runs it to produce the profiling file, and examines the result.

cc -p -o index.assist index.assist.c
index.assist
prof index.assist

The flat output ranks by time across six columns: percentage of total runtime, seconds, cumulative seconds, number of calls, milliseconds per call, and the routine name. In the run recorded in the same chapter, compare_strings consumed 19.4 per cent of total runtime and the _strlen library routine 15.6 per cent, which is enough to rank the work and too little to choose the fix. The flat profile leaves out the two things that decide the fix:

• whether the time came from one expensive call or from a great many cheap ones;
• who called the routine, and whether that caller was itself running in a loop.

The gprof call graph answers both, and the worked example shows why the answer changes the work. The profiling tools chapter shows that when compiled with the call-graph option and analysed through gprof, insert_index_entry is called 760 times from main and calls itself 10,392 times, for 11,152 invocations in total; compare_entry is called 11,152 times, one per invocation, which is what the recursion accounts for; and insert_page_entry is called 820 times, 761 of them from main while the index is built and 59 from insert_index_entry, with those 59 duplicate entries freed once their page data are linked onto the node. A discrepancy in that arithmetic is how a logic bug announces itself, which is a second use for a call graph that the flame graph never quite gives you.

Granularity is the fundamental basis for all of it. The header of that gprof output, quoted in the profiling tools chapter, reports that each sample hit covers four bytes and 0.14 per cent of a 14.74 second run, so the call counts are exact where instrumentation recorded them while the times remain sampled estimates. Statement-level analysis goes one level further: The program compiled with the coverage option maintains a database of basic blocks that is updated on every exit, and tcov turns it into an annotated copy of the source with an execution count next to each line. The summary tcov produces at the end of the file, reproduced in the same chapter, reports 55 of 77 basic blocks executed, rendered as 71.43 per cent of the file, and coverage numbers measure the test input. It also contains an operational fact worth keeping: the coverage database is updated through a single lock file, only one instrumented executable should run on the system at a time, and a manually killed run leaves the lock behind for the next one to trip over.

Instrumentation costs you the run you wanted to measure

Microsoft's Visual Studio tooling is built for the other half of the trade. The overview of the profiling tools explains that the tools in the Performance Profiler are intended to provide analysis for Release builds, that you collect diagnostic information while the application runs and examine it afterwards as post-mortem analysis, and that the Instrumentation tool gives exact call counts and wall-clock time where the CPU Usage tool gives CPU utilisation. Instrumentation requires more overhead than sampling. That is the same trade as the command-line case, arranged as a menu.

The debugger-integrated path is located next to it. The Diagnostic Tools window opens with Ctrl + Alt + F2, the CPU Usage tool arms from the Summary view, and the practical technique is to set a breakpoint at each end of the region you care about and read the data at the second one. PerfTips report the elapsed time between one stepping operation and the next, which times a single function without a separate profiling build. The same overview notes that the profiler can open traces collected elsewhere and that dotnet-trace produces sampling results, with instrumentation left to the profiler itself. .NET Counters arrived in Visual Studio 2019 version 16.7 and BenchmarkDotNet data became viewable in the profiling tools from Visual Studio 2022 version 17.9. Both halves of that trade measure the same thing: where execution stopped, or where an instrumented call occurred. Neither of them measures the hardware state that made the work slow.

What a profile does not show

A profiler reports where its samples landed. Why the CPU was busy at those addresses is a separate measurement, and the useful readings come from leaving the profiler for the hardware counters.

The Red Hat App Services performance team's walkthrough, Profiler Performance Engineering: A Story of What It Does and Doesn't Show, runs that measurement against a Java service under load. perf stat on the process reported 12.570 CPUs utilised and 0.64 instructions per cycle across a 700-second interval. An IPC of 0.64 says the CPU spent most of its cycles waiting for data. The cache-miss flame graph shows the widest bar over java.util.HashMap$KeyIterator.next, and its account of the mechanism is what the profiler alone could not supply: iterating a HashMap walks a chain of scattered nodes, each one holding the address of the next, so the hardware prefetcher cannot run ahead and the pipeline stalls between elements. Confirmation comes from comparing the miss profile with the cycles profile and finding the same methods at the top of both. The fix belongs in the data structure. The same walkthrough frames the exercise with the USE method, which it attributes to Brendan Gregg: for each resource in turn, ask about utilisation, saturation and errors, and let the answers direct which profiler to open.

The mechanism determines the reading. Sampling estimates where the time went and costs almost no overhead, which is why it survives contact with production. Deterministic tracing counts every call and changes the run it measures, which is why it belongs on a bench with a fixed workload. Instrumentation gives exact call counts and wall-clock time at a cost the Visual Studio documentation explains plainly, and the hardware counters answer the question neither of them asks. Pick the mechanism from the question, state the question before the run, and treat every profile as a reading taken by an instrument, with the instrument's bias included.

Sources

• Performance Profiling Tools (Sun compiler documentation, hosted by the University of Edinburgh's School of Informatics)
• profiling — Python profilers (Python 3.15 documentation)
• Overview of the profiling tools (Microsoft Learn)
• Profiler Performance Engineering: A Story of What It Does and Doesn't Show (Red Hat App Services performance team)