Slow Performance - Windows Performance Counter API

I recently had the pleasure opportunity to work with the Windows Performance Counter API as a data store for a metrics-type service in ASP.NET Web API. We needed to generate a large number of counters (in the thousands) for several partners and properties, and used them to provide data endpoints that partners could consume.

Adding the first couple of partners was a breeze, but as we rolled out the service to more partners, performance quickly began to degrade and our beautiful service became unusable. Requests took minutes to process, and timeouts were rampant. But the Windows Performance Counter system is supposed to be fast, right?

I couldn’t see any bottlenecks in my code, save for the actual Performance Counter API calls. There was a slight delay in getting all counters for a category, as expected, but reading each counter took anywhere from 10-20ms, and the per-counter read time worsened with the addition of more counters. For several thousand counters, that’s a lot of lag! Let’s also consider that many counters are read twice to collect average or timed data.

The Problem

Category overload: To dig deeper into the problem, I created a counter Category for each partner (instead of putting all counters in a single category), which immediately seemed to help. This brought the per-counter read time down drastically. So it was clear that the read time per counter was affected by the total number of counters in a category–an exponential problem. Strangely, even iterating through a small subset of counters from an overloaded category wasn’t much better.

Here are some stats when reading counter values from a single category:

• 8 total counters, 1-2ms per counter
• 256 total counters, 15-18ms per counter
• 512 total counters, 30ms per counter
• 3,584 total counters (reading all counters), 200ms per counter
• 3,584 total counters (reading only 512 counters), 50-90ms per counter.

I used System.Diagnostics.Stopwatch before and after the change to verify this:

Stopwatch t;
foreach (var c in counters)
{
    t = Stopwatch.StartNew();
    var r = c.RawValue;
    Debug.WriteLine(t.ElapsedMilliseconds.ToString("000") + " - " + c.CategoryName + ":" + c.CounterName + "(" + c.CounterType + ") = " + r);
}

The Solution

More categories! That’s it. Each partner now has its own category, each having 8-10 counters. Read time is < 5ms total, and everyone is happy. To test, I was able to programmatically spawn a hundred new categories, each having 5-10 counters, and performance was still blazing fast. Crisis averted.

One last thing: When working with the Performance Counter API, the addition of more than just a few counters will require either a registry change or a machine.config update to allocate more memory to the process. More information can be found here: MSDN - performanceCounters Element

Relevant links

• StackOverflow Question
• MSDN - performanceCounters Element