Currently, our ServiceNow SolarWinds integration is functioning to an extent but there is one major issue with it.
It changes State values in most cases as it should but I've been able to replicate our issue by doing the following
I bring node down, creates an incident if there isn't one
I bring Node back up, sets incident state to monitoring
I bring the Node back down, sets incident state to active
I bring Node back up, sets incident state to monitoring
Everything is functioning fine up until this point
So lets say we have verified functionality and their monitoring period is up
I set the incident state to resolved (value 6) within ServiceNow
I bring the Node back down, it sends over work notes but it does not send a state value with it
the payload within the Incident Integrations module shows
{"sys_id":"<incident sys_id>","work_notes":"Orion alert has been reactivated."}
with <incident sys_id> being the incidents sys id
I figured that it might be an issue with the orion integration account not being able to update incidents in a resolved state
to test it, i attempt to bring the node back up, it changes the state just fine on over to monitoring
I figure maybe its an issue with solarwinds attempting to update state immediately following someone manually updating state on the incident (maybe it throws it into an acknowledged state and following that, it doesnt update state)
Made sure that Do not execute this action if the alert has been acknowledged already (Recommended) was unchecked so that it will continue to update the incident following (even though it seems to update fine on node going back up using the same action)
Attempted to see if I could get it to change state from resolved immediately following manual resolve using same action
Brought node down, sets state to active
Manually change state to resolved in ServiceNow
Brought node back up, sets state to Monitoring successfully
Issue is with Reopen state specifically following a manual change within ServiceNow to Resolved
Reopen state works otherwise and if i bring node back up and back down, it will set state normally.
It is only immediately following doing a manual update changing the state to resolved that it does not send over an updated state value
I had thought that maybe it could be the Acknowledge State that is playing with it a bit but i set the Acknowledge State to active from resolved and was still getting the exact same issues
I ran a search within the incident integrations module to see if it ever was able to update the state from resolved to something else using that same action
query goes something like this
comment contains state: "6"
comment does not contain incident_state: "6"
payload contains Orion alert has been reactivated
I get 10 results but these are for things running different actions
In the actions, instead of setting incidents to monitoring, they set them to resolved when node goes back up
When they go back down, it is updating to active just fine meaning that its not an issue going from resolved to active using the Reopen state field
So the issue is narrowed down to,
following a manual modification of the state field, Reopen state fails to include state key value pair in the message it sends to update the incidents state
Any assistance would be greatly appreciated with this issue!
We opened a case number with support a few weeks back and recently, they suggested posting here for assistance along with including the case number
This is a huge issue because when nodes go back down following being resolved, no one is being notified because the state is not changing back to active
