Hi
My additional pollers cannot collect data, or I should say the data is discontinuous. But my primary poller works well.
Like this:
I find logs on my primary poller servers:
*** SolarWinds Job Engine v1.6.0.28, .Net Runtime v4.0.30319 ***
2014-03-08 10:45:11,280 [WorkerProcessMonitor] INFO SolarWinds.JobEngine.Engine.WorkerMonitor - Job monitoring starting
2014-03-08 10:45:11,296 [6] INFO SolarWinds.JobEngine.PluginStore - Refreshing using C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins as plug-ins manifest container
2014-03-08 10:45:11,311 [6] INFO SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.Core.Discovery.Jobs.plugin"
2014-03-08 10:45:11,327 [6] INFO SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.Core.Jobs.plugin"
2014-03-08 10:45:11,327 [6] INFO SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.IpSla.Jobs.plugin"
2014-03-08 10:45:11,327 [6] INFO SolarWinds.JobEngine.PluginStore - Processing plug-in file "C:\Program Files (x86)\Common Files\SolarWinds\JobEngine\plugins\SolarWinds.Orion.IpSla.Jobs.SNMP.plugin"
2014-03-08 10:45:16,475 [6] INFO SolarWinds.JobEngine.ServiceProcess.SolarWindsService - OnStart: Application started
2014-03-08 10:46:19,459 [QueueMonitoringManager] INFO SolarWinds.JobEngine.Engine.WorkerProcessWCFProxy - worker process 78b4a84d-5ad7-47ac-aae1-c99d14e33087 has pid 11140
2014-03-08 10:50:15,342 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-08 23:10:17,601 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-08 23:15:17,676 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 03:40:18,739 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 03:45:18,740 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 03:50:18,878 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 03:55:18,941 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 14:30:20,295 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 15:50:20,766 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:20:24,058 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:25:24,576 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:30:25,512 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:35:25,851 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:40:25,992 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:45:26,059 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:50:26,087 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 17:55:26,108 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:00:26,130 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:05:26,150 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:10:26,171 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:15:26,192 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:20:26,213 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:25:26,234 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:30:26,255 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:35:26,276 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:40:26,297 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:45:26,318 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:50:26,339 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 18:55:26,360 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:00:26,382 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:05:26,402 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:15:27,834 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:20:27,855 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:25:27,876 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:30:27,897 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:35:27,824 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:40:27,756 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:45:27,892 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:50:27,961 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 19:55:28,000 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:00:28,021 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:05:28,041 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:10:28,062 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:15:28,083 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:20:28,104 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:25:28,125 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:30:28,146 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:35:28,167 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:40:28,188 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:45:28,209 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:50:28,230 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 20:55:28,251 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:00:28,272 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:05:28,294 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:10:28,315 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:15:28,336 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:20:28,357 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:25:28,378 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:30:28,399 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:35:28,420 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:40:28,441 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
2014-03-09 21:45:28,462 [ScheduledOperations] ERROR SolarWinds.JobEngine.ScheduledOperationManager - Woke up too early!
What is 'Woke up too early' ?!
Other logs on my primary poller:
2014-03-08 10:45:18,191 [6] INFO SolarWinds.JobEngine.Scheduling.SchedulerServiceApp - Database versions match: 2
2014-03-08 10:45:18,284 [6] INFO SolarWinds.JobEngine.Scheduling.CredentialHandlerFactory - Using certificate for credentials protection
2014-03-08 10:45:18,362 [6] INFO SolarWinds.JobEngine.Broker.JobExecutionEngineManager - Wait period between retry attempts will be: 300 seconds
2014-03-08 10:45:18,394 [6] INFO SolarWinds.JobEngine.Broker.ActiveJobJournal - Initialized with 0 active jobs
2014-03-08 10:45:22,621 [6] INFO SolarWinds.JobEngine.Broker.ActiveJobJournal - Retrieved 0 incomplete jobs
2014-03-08 10:45:23,838 [6] INFO SolarWinds.JobEngine.ServiceProcess.SolarWindsService - OnStart: Application started
2014-03-08 10:46:12,699 [14] INFO SolarWinds.JobEngine.Broker.JobEngineWCFProxy - job engine 86547d36-4cce-43e4-86e6-1bf994a47e54 located at net.pipe://poller001/solarwinds/jobengine/engine
2014-03-08 10:46:25,045 [ResultDispatcher] WARN SolarWinds.JobEngine.Scheduling.JobScheduler - Notification uri not found [net.pipe://localhost/orion/ipsla/servicemonitor/echo]. Next 100 endpoint not found will be suppressed
2014-03-08 10:46:26,061 [ResultDispatcher] WARN SolarWinds.JobEngine.Scheduling.JobScheduler - Notification uri not found [net.pipe://localhost/orion/ipsla/servicemonitor/echo]. Next 100 endpoint not found will be suppressed
2014-03-08 10:47:04,604 [5] INFO SolarWinds.JobEngine.Broker.JobEngineWCFProxy - job engine f7c98c62-d254-4d09-bee4-287d9033f084 located at net.tcp://poller002:17777/solarwinds/jobengine/engine
2014-03-08 10:47:05,624 [5] INFO SolarWinds.JobEngine.Broker.JobEngineWCFProxy - job engine f7c98c62-d254-4d09-bee4-287d9033f084 located at net.tcp://poller002:17777/solarwinds/jobengine/engine
2014-03-08 10:47:06,663 [5] WARN SolarWinds.JobEngine.Broker.JobEngineInfo - Error reseting proxy
SolarWinds.JobEngine.Broker.JobEngineNotListeningException: The job engine f7c98c62-d254-4d09-bee4-287d9033f084 is not listening ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://poller002:17777/solarwinds/jobengine/engine/ssl. The connection attempt lasted for a time span of 00:00:01. TCP error code 10061: No connection could be made because the target machine actively refused it 10.x.x.3:17777. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.x.x.3:17777
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedFramingRequest.SendRequest(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open()
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Open()
at SolarWinds.JobEngine.Broker.JobEngineWCFProxy.EstablishConnection(MessageCredentialType credentialType)
at SolarWinds.JobEngine.Broker.JobEngineWCFProxy.EstablishConnection(MessageCredentialType credentialType)
at SolarWinds.JobEngine.Broker.JobEngineWCFProxy.Connect()
at SolarWinds.JobEngine.Broker.JobEngine.Reset()
--- End of inner exception stack trace ---
at SolarWinds.JobEngine.Broker.JobEngine.Reset()
at SolarWinds.JobEngine.Broker.JobEngineInfo.Reconnect()
Is there anybody know how to solve it?
Thanks in advance.
/Alex