Benchmarking Preparation Checklist

In gathering performance metrics on a piece of software, you will want to spend some amount of effort in minimizing the noise in the system (reducing the standard deviation on the numbers collected). Here are some actions which may be beneficial in giving you tighter control of your environment.

System Preparation

• Remove any extraneous peripherals. Additional peripherals may cause hardware based interrupts to fire which can interrupt the application you are attempting to profile.
• Disconnect from the network. Incoming network requests from other systems can cause spurious events to occur. If you need network access, you may want to disconnect from a Domain which has the ability to run various tasks in the background (without your knowledge).
• Disable or pause any unnecessary services. You can programmatically pause specific services using OpenService then calling ControlService with SERVICE_CONTROL_PAUSE.
• Disable UI effects. Calling SystemParametersInfo will let you disable (and then restore) miscellaneous UI elements.

• Turn off the screen saver: SPI_SETSCREENSAVEACTIVE
• Minimize popup durations: SPI_SETMESSAGEDURATION
• Turn off blinky animations: SPI_SETCLIENTAREAANIMATION
• Remove drop shadows: SPI_SETDROPSHADOW
• Enable flat menus: SPI_SETFLATMENU
• Disable font smoothing: SPI_SETFONTSMOOTHING
• Remove the mouse shadow: SPI_SETCURSORSHADOW

Process Preparations

• Flush caches. You will want to ensure that the benchmarks start from a consistent system state. One approach is to start with all caches warmed, the other is from a cold state. For the latter, here are some APIs which can help clean things up for you, assuming you don't have the time to do a full cold reboot:

• Empty clipboard (EmptyClipboard)
• Flush registry (RegFlushKey)
• Flush file volumes (FlushFileBuffers on each volume e.g. "\\.\c:")
• The OS's file system cache (SetSystemFileCacheSize with -1)

• Close or minimize any other applications. Any process which is running may have the chance to interrupt you and aside from the time lost, it will destroy any cache coherency you've built up.
• Move all remaining processes onto a single CPU core. By isolating all the remaining processes onto core 0 (for example), we can prevent as much disruption to your process as possible. You may want to run under the SYSTEM account, otherwise you won't have permission to move some processes via SetProcessAffinityMask.
• Drop priority on any above-normal processes. We will want to run at real-time, so let us make sure that nobody else has a chance to compete for time slices (SetProcessPriorityBoost and SetPriorityClass).

Running

• Set the processor affinity. Run your application on a core which it has all to itself.
• Set priority to real-time. Max out the scheduler priority so it is less likely to be interrupted.
• Capture the pipes to memory. If you are profiling a console application, piping the output to a file will probably defeat your performance metrics. Instead stick the data into a memory buffer (assuming you are launching the application under test via CreateProcess).

Afterwards

• Use GetProcessTimes, QueryPerformanceCounter, and QueryProcessCycleTime to gather baseline timing metrics.
• You may want to call GetProcessIoCounters or GetProcessMemoryInfo to retrieve additional data.
• Be sure to restore the system changes made in the preparations steps.