This morning a member of our team was looking at processor load over time on a windows 2008 R2 host and noted that what he saw in SolarWinds NPM did not match what he was seeing when he looked at the system's load through Task Manager and later Perfmon on the system The data in NPM was consistently higher. He asked me about this and I did a bit of digging and found a disturbing trend in the source data within SolarWinds.
In short - NPM appears to be recording the same value for min and max CPU (and hence average CPU) in the CPUMultiLoad_Detail, CPUMultiLoad, and CPULoad tables (my guess would be that the latter is derived from the former). I am guessing that this is the result of one of the following:
- Stupid-Human-Trick 1 - I'm looking at the wrong tables in the database (but the data I see in the table aligns with what I am seeing in the graphs which are out of line with the graphs that we are seeing in the other sources - so I doubt this is the case)
- Stupid-Human-Trick 2 - I made a mistake on installation which resulted in this averaging before population (possible - although I was pretty careful during the portion of the configuration in which actual decisions were being made).
- Fixed Bug - There is a known issue in the software - apply the update... (note - I did do a bit of digging and did not find this listed
- Active Bug - There is a known bug that is not resolved
- Other?
My money is on the second or third option - but searching for a few did not provide any information on this symptom. Data and system version information below...
Thanks in advance...
DATA / SYSTEM INFORMATION
For example - the data for the node in question for the last couple of windows from each table follows:
FROM CPUMultiLoad
NodeID | TimeStampUTC | CPUIndex | MinLoad | MaxLoad | AvgLoad | Weight |
326 | 2015-01-12 22:13:54.363 | 0 | 12 | 12 | 12 | 600 |
326 | 2015-01-12 22:13:54.363 | 1 | 12 | 12 | 12 | 600 |
326 | 2015-01-12 22:13:54.363 | 2 | 0 | 0 | 0 | 600 |
326 | 2015-01-12 22:13:54.363 | 3 | 6 | 6 | 6 | 600 |
326 | 2015-01-12 22:13:54.363 | 4 | 0 | 0 | 0 | 600 |
326 | 2015-01-12 22:13:54.363 | 5 | 18 | 18 | 18 | 600 |
326 | 2015-01-12 22:13:54.363 | 6 | 6 | 6 | 6 | 600 |
326 | 2015-01-12 22:13:54.363 | 7 | 18 | 18 | 18 | 600 |
326 | 2015-01-12 22:03:54.363 | 0 | 6 | 6 | 6 | 600 |
326 | 2015-01-12 22:03:54.363 | 1 | 93 | 93 | 93 | 600 |
326 | 2015-01-12 22:03:54.363 | 2 | 93 | 93 | 93 | 600 |
326 | 2015-01-12 22:03:54.363 | 3 | 31 | 31 | 31 | 600 |
326 | 2015-01-12 22:03:54.363 | 4 | 6 | 6 | 6 | 600 |
326 | 2015-01-12 22:03:54.363 | 5 | 37 | 37 | 37 | 600 |
326 | 2015-01-12 22:03:54.363 | 6 | 93 | 93 | 93 | 600 |
326 | 2015-01-12 22:03:54.363 | 7 | 100 | 100 | 100 | 600 |
FROM CPUMultiLoad Detail
NodeID | TimeStampUTC | CPUIndex | MinLoad | MaxLoad | AvgLoad |
326 | 2015-01-12 23:03:54.397 | 0 | 0 | 0 | 0 |
326 | 2015-01-12 23:03:54.397 | 1 | 37 | 37 | 37 |
326 | 2015-01-12 23:03:54.397 | 2 | 93 | 93 | 93 |
326 | 2015-01-12 23:03:54.397 | 3 | 93 | 93 | 93 |
326 | 2015-01-12 23:03:54.397 | 4 | 12 | 12 | 12 |
326 | 2015-01-12 23:03:54.397 | 5 | 93 | 93 | 93 |
326 | 2015-01-12 23:03:54.397 | 6 | 93 | 93 | 93 |
326 | 2015-01-12 23:03:54.397 | 7 | 49 | 49 | 49 |
326 | 2015-01-12 22:53:54.397 | 0 | 0 | 0 | 0 |
326 | 2015-01-12 22:53:54.397 | 1 | 0 | 0 | 0 |
326 | 2015-01-12 22:53:54.397 | 2 | 47 | 47 | 47 |
326 | 2015-01-12 22:53:54.397 | 3 | 64 | 64 | 64 |
326 | 2015-01-12 22:53:54.397 | 4 | 0 | 0 | 0 |
326 | 2015-01-12 22:53:54.397 | 5 | 88 | 88 | 88 |
326 | 2015-01-12 22:53:54.397 | 6 | 58 | 58 | 58 |
326 | 2015-01-12 22:53:54.397 | 7 | 41 | 41 | 41 |
FROM CPULoad
NodeID | DateTime | MinLoad | MaxLoad | AvgLoad | TotalMemory | MinMemoryUsed | MaxMemoryUsed | AvgMemoryUsed | AvgPercentMemoryUsed | Archive | Weight |
326 | 2015-01-12 15:13:54.363 | 9 | 9 | 9 | 6.012908E+10 | 3.840328E+09 | 3.840328E+09 | 3.840328E+09 | 6.386806 | 0 | 600 |
326 | 2015-01-12 15:03:54.363 | 57 | 57 | 57 | 6.012908E+10 | 3.835929E+09 | 3.835929E+09 | 3.835929E+09 | 6.37949 | 0 | 600 |
SO - The values in the MinLoad, MaxLoad, and AvgLoad fields in the CPULoad table are each the same, and each the rounded average of the full set of values for all processors in the node for the same window in CPUMultiLoad. I have to assume that there is an issue with my configuration that is causing this behavior (or at least I HOPE that this is not the intended behavior as it certainly is not "good data" regarding min/max/avg CPU load since the last poll.
Am I looking at the wrong data in the database? What causes this?
Environment Details:
FROM SETTINGS: Orion Platform Details (redacted a couple of items not germane to the question - like the hostname)
Orion Platform Details | |
Server | |
Web Server | |
Operating System | Windows Server 2008 R2 |
OS Version | 6.1.7601.65536 |
Service Pack | Service Pack 1 |
Network Elements | |
Total Count | |
Nodes | |
Interfaces | |
Volumes | |
Version Info | |
AccountLimitationBuilder.exe | 2014.01 |
AlertingEngine.exe | 2014.1.0.0 |
AlertManager.exe | 2014.01 |
AntiXSSLibrary.dll | 4.2.0.0 |
Antlr3.Runtime.dll | 3.1.0.39271 |
Antlr3.Utility.dll | 0.1.0.39272 |
BasicAlertsManager.exe | 2014.01 |
ConfigurationWizard.exe | 2014.1.4.1466 |
Database-Maint.exe | 2014.1.4.1466 |
DatabaseManager.exe | 2014.1.4.1466 |
EO.Pdf.dll | 5.0.34.2 |
ExportToPDFCmd.Exe | 2014.1.4.1466 |
HtmlSanitizationLibrary.dll | 4.2.0.0 |
ICSharpCode.SharpZipLib.dll | 0.85.5.452 |
Infragistics2.Shared.v7.3.dll | 7.3.20073.38 |
InstallHelper.RunOnce.exe | 2014.1.4.1466 |
Interop.cdosys.dll | 1.0.0.0 |
Interop.COMAdmin.dll | 1.0.0.0 |
Interop.NetFWTypeLib.dll | 1.0.0.0 |
Interop.olelib.dll | 1.61.0.0 |
Interop.OrionSWScheduler8.dll | 1.0.0.0 |
Interop.SHDocVw.dll | 1.1.0.0 |
Interop.TaskScheduler.dll | 1.0.0.0 |
Kiwi_LogViewer_2.1.0_Win32.setup.exe | Unknown |
LicenseManager.exe | 3.0.0.528 |
log4net.dll | 1.2.10.0 |
LogAdjuster.exe | 2014.1.4.1466 |
makecert.exe | 5.131.3790.0 (srv03_rtm.030324-2048) |
Microsoft.Web.Services3.dll | 3.0.5305.0 |
MomConnectorFrameworkV2.dll | 5.0.2749.0 |
NetPerfMonEventLog.dll | 6.0.0.0 |
nsoftware.IPWorksSSNMP.dll | 9.0.5077.0 |
OptimizeWebsite.exe | 2014.1.4.1466 |
OrionAlertEngine.exe | 2014.01 |
OrionNPMMapEngine.dll | 2014.01 |
OrionNPMWeb.dll | 2014.01 |
OrionPermissionChecker.exe | 2014.1.4.1466 |
OrionReportWriter.exe | 2014.01 |
OrionSWDiscoveryServices9.dll | 2014.01 |
OrionSWLogger.dll | 2014.01 |
OrionSWScheduler8.dll | 2014.01 |
OrionWeb.dll | 2014.1.4.1466 |
OrionWebBackup.exe | 2014.01 |
phantomjs.exe | 1.9.2 |
PollerCheckerTool.exe | 2014.1.4.1466 |
PrintLauncher.exe | 2014.1.4.1466 |
RegAsm.exe | 4.0.30319.1 (RTMRel.030319-0100) |
SciLexer.dll | 3.0.4 |
SciLexer64.dll | 3.0.4 |
ScintillaNet.dll | 2.5.0.0 |
SmartThreadPool.dll | 2.0.0.0 |
SnmpWalk.exe | 2011.2.0.0 |
SolarWinds.BusinessLayerHost.Contract.dll | 2014.1.4.1466 |
SolarWinds.BusinessLayerHost.exe | 2014.1.4.1466 |
SolarWinds.Cirrus.IS.Client.dll | 7.3.3.897 |
SolarWinds.Collector.Contract.dll | 2.9.0.72 |
SolarWinds.Common.dll | 2014.1.4.1466 |
Solarwinds.Compression.dll | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.Common.dll | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.IIS7.dll | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.IIS7.Installer.exe | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.Permissions.dll | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.Plugin.APM.dll | 6.1.1.3047 |
SolarWinds.ConfigurationWizard.Plugin.Common.dll | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.Plugin.IpSla.dll | 4.1.0.1568 |
SolarWinds.ConfigurationWizard.Plugin.NPM.dll | 10.7.4.810 |
SolarWinds.ConfigurationWizard.Plugin.OIP.dll | 2.1.113.0 |
SolarWinds.ConfigurationWizard.Plugin.Orion.dll | 2014.1.4.1466 |
SolarWinds.ConfigurationWizard.Plugin.VIM.dll | 1.9.0.69 |
SolarWinds.DatabaseManager.Strings.dll | 2014.1.4.1466 |
SolarWinds.DataGap.AnalysisTool.exe | 2014.1.4.1466 |
SolarWinds.Diagnostics.DBResponse.exe | 2014.1.4.1466 |
SolarWinds.FipsManager.exe | 2014.1.4.1466 |
SolarWinds.InformationService.Contract2.dll | 2014.1.0.45 |
SolarWinds.Internationalization.dll | 2014.1.4.1466 |
SolarWinds.JobEngine.Contract.dll | 1.6.0.28 |
SolarWinds.JobEngine.Contract2.dll | 2.7.0.30 |
SolarWinds.Licensing.Analysis.dll | 3.0.0.528 |
SolarWinds.Licensing.Diagnostics.exe | 3.0.0.528 |
SolarWinds.Licensing.Framework.dll | 3.0.0.528 |
SolarWinds.Logging.dll | 2014.1.4.1466 |
SolarWinds.Management.dll | 2014.1.4.1466 |
SolarWinds.NCM.AlertAction.dll | 7.3.3.897 |
SolarWinds.NCM.BusinessLayerPlugin.dll | 1.0.1.0 |
SolarWinds.NCM.Contracts.dll | 7.3.3.897 |
SolarWinds.NCM.RTNForwarder.exe | 7.3.3.897 |
SolarWinds.Net.ICMP.dll | 2014.1.4.1466 |
SolarWinds.Net.SNMP.dll | 2014.1.4.1466 |
SolarWinds.Net.WMI.dll | 2014.1.4.1466 |
SolarWinds.NTM.Common.dll | 1.0.0.1 |
SolarWinds.NTM.Data.dll | 1.0.0.1 |
SolarWinds.NTM.Discovery.dll | 1.0.0.1 |
SolarWinds.NTM.Topology.dll | 1.0.0.1 |
SolarWinds.Orion.Collector.dll | 2014.1.4.1466 |
SolarWinds.Orion.Collector.MessagePlugin.dll | 2014.1.4.1466 |
SolarWinds.Orion.Collector.Plugins.dll | 2014.1.4.1466 |
SolarWinds.Orion.Common.dll | 2014.1.4.1466 |
SolarWinds.Orion.CommonHelper.Interop.dll | 1.0.0.0 |
SolarWinds.Orion.Core.Actions.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Alerting.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Auditing.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.BusinessLayer.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Collector.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Collector.MessageSender.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Common.dll | 2014.1.4.1466 |
Solarwinds.Orion.Core.Data.dll | 2014.1.4.1466 |
Solarwinds.Orion.Core.Dependencies.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Discovery.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Jobs.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Jobs2.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.MessageActions.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Models.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Pollers.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Reporting.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.SharedCredentials.dll | 2014.1.4.1466 |
SolarWinds.Orion.Core.Strings.dll | 2014.1 |
SolarWinds.Orion.Core.SwisViaCOM.dll | 1.0.0.0 |
SolarWinds.Orion.Core.Topology.dll | 2014.1.4.1466 |
SolarWinds.Orion.Discovery.Contract.dll | 2014.1.4.1466 |
SolarWinds.Orion.Discovery.Framework.dll | 2014.1 |
SolarWinds.Orion.Discovery.Job.dll | 2014.1 |
SolarWinds.Orion.FeatureManager.Interop.dll | 1.0.0.0 |
SolarWinds.Orion.I18n.Interop.dll | 1.0.0.0 |
SolarWinds.Orion.MacroProcessor.dll | 2014.1.0.0 |
SolarWinds.Orion.NCM.Security.dll | 7.3.3.897 |
SolarWinds.Orion.Packages.dll | 1.0.0.55 |
SolarWinds.Orion.Pollers.Framework.dll | 2014.1.4.1466 |
SolarWinds.Orion.Security.dll | 2014.1.4.1466 |
SolarWinds.Orion.Topology.Calculator.exe | 2014.1.4.1466 |
SolarWinds.Orion.Topology.Framework.dll | 2014.1.4.1466 |
SolarWinds.OrionScheduler2.dll | 2014.1.4.1466 |
SolarWinds.Packaging.Contract.dll | 1.0.0.55 |
SolarWinds.Packaging.Core.dll | 1.0.0.55 |
SolarWinds.PollerChecker.Contract.dll | 2014.1.4.1466 |
SolarWinds.PollerChecker.dll | 2014.1.4.1466 |
SolarWinds.PollerChecker.Plugin.Core.dll | 2014.1.4.1466 |
SolarWinds.PollerChecker.Plugin.PollerChecker.dll | 2014.1.4.1466 |
SolarWinds.Reporting.dll | 2014.1.4.1466 |
SolarWinds.Security.Cryptography.X509Certificates.dll | Unknown |
SolarWinds.Syslog.dll | 2013.2.0.1 |
SolarWinds.Syslog.Strings.dll | 2013.2.0.1 |
SolarWinds.WebLauncher.exe | 2014.1.4.1466 |
SolarwindsDiagnostics.exe | 2014.1.4.1466 |
StopNetPerfMon.exe | 2014.1.4.1466 |
SWAlertActions8.dll | 2014.01 |
SWAlertEngine8.dll | 2014.01 |
SWSubClass.dll | 2014.01 |
SWSysLogActions.dll | 2014.01 |
SWTrapActions.dll | 2014.01 |
SWTrapService.exe | 2014.1.4.1466 |
SyslogService.exe | 2014.1.4.1466 |
SyslogViewer.exe | 2014.01 |
taskschd.dll | 6.0.6001.18000 (longhorn_rtm.080118-1840) |
tdbgpp8.dll | 8.0.20034.14 |
TDBWSnk6.dll | 5.00.0020 |
TestAlerts.exe | 2014.01 |
todgub8.dll | 8.0.20034.18 |
TrapViewer.exe | 2014.01 |
Vim25Service.dll | 0.0.0.0 |
Vim25Service.XmlSerializers.dll | 0.0.0.0 |
VimService.dll | 0.0.0.0 |
VimService.XmlSerializers.dll | 0.0.0.0 |