When adatole posted his The Ultimate CPU Alert we all cheered. His alert successfully paired CPU utilization and CPU queue length to accurately measure true load on a Windows server. His alert successfully unlocked the mystery of CPU utilization to a whole generation of Thwacksters. Scores of administrators implemented the SAM template and custom SQL alert logic and rejoiced. There was, however, an Achilles heel. The query relies on CPU count -- a statistic that is available with the data in the Orion 2014.1 version of the DB but requires that it be calculated.
NOTE: The queries posted below use a couple of node custom properties. You should create a n_CPUCount but do not populate it and, if you want to use the queries as pasted below, a n_CPUCrit which is a whole number threshold between 1 and 100 for when you want the CPU alert to trigger. The logic below assumes a 95% threshold if n_CPUCrit isn't populated.
When ciulei started digging through our Database Performance Analyzer instance he noticed that the SQL logic for The Ultimate CPU Alert kept showing up as one of the top queries that was generating wait time. Our DB server had recently been moved to a brand new box with 256GB RAM and a 4 x SSD RAID10 array to support our 17 additional pollers and 11,000 nodes/72,000 elements. It didn't look like a hardware problem so we decided to jump into the SQL logic of the query. After brainstorming with ciulei and adatole we came up with a plan of attack.
The foundation of the plan was pre-calculating the CPUCount for the query using a stored procedure in our DB. This is what ciulei came up with for us. It calculates the CPU count (note the +1 since CPUIndex starts with 0) and writes that value into the nodes table to a custom property called n_CPUCount. Create the custom property first then schedule the stored procedure to run at an interval that suits your environment. We run our procedure once per day and we only check for data in the last 24 hours.
SELECTDISTINCT NodeID
,MAX(CPUIndex)+1 AS CPUCount
INTO #CPUCount
FROM CPUMultiLoad
WHEREDATEDIFF(hh,TimeStampUTC,GETDATE())<= 24
GROUPBY NodeID;
UPDATE n
SET n.n_CPUCount = t.CPUCount
FROM Nodes n
JOIN #CPUCOUNT t ON n.NodeID = t.NodeID;
DROPTABLE #CPUCount;
To take advantage of the new nodes.n_CPUCount you will need to modify both the trigger and reset queries for the Windows CPU alert. Note that we forced a check for n_CPUCount IS NOT NULL. Why? We have some nodes (usually network hardware) that is accessible via SNMP but doesn't not provide CPU statistics. We're working on resolving those with custom pollers that we'll assign to specific hardware but without CPU data in the CPUMultiLoad table there is no way to gather the number of CPUs.
Trigger Query
SELECTNodes.NodeIDASNetObjectID,Nodes.CaptionASName
FROMNodes
INNERJOINAPM_AlertsAndReportsDataon (Nodes.NodeID=APM_AlertsAndReportsData.NodeId)
WHERE
- Nodes.n_mute<> 1
ANDNodes.Prod_State='PROD'
ANDAPM_AlertsAndReportsData.ComponentName='Win_Processor_Queue_Len'
ANDNodes.n_CPUCountISNOTNULL
ANDAPM_AlertsAndReportsData.StatisticData>nodes.n_CPUCount
AND
(
(nodes.CPU_Critisnull
ANDnodes.CPULoad>= 90)
OR(nodes.CPU_Critisnotnull
ANDnodes.CPULoad>=nodes.CPU_Crit)
)
Reset Query
SELECTNodes.NodeIDASNetObjectID,Nodes.CaptionASName
FROMNodes
INNERjoinAPM_AlertsAndReportsDataon (Nodes.NodeID=APM_AlertsAndReportsData.NodeId)
WHEREnodes.n_mute<> 1
ANDNodes.Prod_State='PROD'
ANDAPM_AlertsAndReportsData.ComponentName='Win_Processor_Queue_Len'
ANDNodes.n_CPUCountISNOTNULL
ANDAPM_AlertsAndReportsData.StatisticData<=Nodes.n_CPUCount
AND
(
(nodes.CPU_Critisnull
ANDnodes.CPULoad< 90)
OR(nodes.CPU_Critisnotnull
ANDnodes.CPULoad<nodes.CPU_Crit)
)
That's it! If you roll out these changes your DB will be much happier. When we implemented the changes to the The Ultimate CPU Alert and The Ultimate CPU Alert ... for Linux! (I'll post the Linux version later) we reduced waits 150% on our DB. Read that again -- we reduced waits by 150% What is the moral of the story? For peak application performance you *must* be watching your database and tuning for performance. If we can do it, so can you!