Have a bit of an interesting issue with some Apple servers we are trying to monitor. All systems are running the default Apple installed net-snmpd 5.4.2.1. How ever on a handful of machines if we poll the CPU the snmpd process crashes and restarts(shows in Orion as a reboot of the system). The snmpd.conf file is identical across all servers. In comparing the hardware of two of the servers, one that has the problem and one that doesn't, but have the exact same OS and hardware. Here is what we get in the system log.
09:42:38.259 ReportCrash[50717:2503] Saved crash report for snmpd[49867] version ??? (???) to /Library/Logs/DiagnosticReports/snmpd_2011-10-03-094238_localhost.crash
Then looking at this crash file we see the following. I have been googling around but not finding much info. Anyone have any ideas. For now I have turned off CPU monitoring on these systems, about a half a dozen. Interesting thing, I have APM doing a process monitor on the AFP service and get CPU info from it.
Process: snmpd [49867]
Path: /usr/sbin/snmpd
Identifier: snmpd
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: launchd [1]
Date/Time: 2011-10-03 09:42:38.007 -0500
OS Version: Mac OS X Server 10.6.8 (10K549)
Report Version: 6
Exception Type: EXC_ARITHMETIC (SIGFPE) Exception Codes: EXC_I386_DIV (divide by zero) Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Thread 0 Crashed: Dispatch queue: com.apple.main-thread
0 libnetsnmpmibs.15.dylib 0x000000010006b437 var_hrproc + 169
1 libnetsnmphelpers.15.dylib 0x000000010004805c
netsnmp_old_api_helper + 390
2 libnetsnmpagent.15.dylib 0x0000000100016f95
netsnmp_call_handlers + 868
3 libnetsnmpagent.15.dylib 0x0000000100009a83 handle_var_requests
+ 91
4 libnetsnmpagent.15.dylib 0x000000010000c179 handle_pdu + 524
5 libnetsnmpagent.15.dylib 0x000000010000d0af
netsnmp_handle_request + 500
6 libnetsnmpagent.15.dylib 0x000000010000d443 handle_snmp_packet +
445
7 libnetsnmp.15.dylib 0x000000010015462f snmp_timeout + 3928
8 libnetsnmp.15.dylib 0x0000000100155605 _sess_read + 3719
9 libnetsnmp.15.dylib 0x00000001001556e5 snmp_read + 30
10 snmpd 0x000000010000284e 0x100000000 + 10318
11 snmpd 0x0000000100000d40 0x100000000 + 3392
Thread 1: Dispatch queue: com.apple.libdispatch-manager
0 libSystem.B.dylib 0x00007fff89f44c0a kevent + 10
1 libSystem.B.dylib 0x00007fff89f46add _dispatch_mgr_invoke
+ 154
2 libSystem.B.dylib 0x00007fff89f467b4
_dispatch_queue_invoke + 185
3 libSystem.B.dylib 0x00007fff89f462de
_dispatch_worker_thread2 + 252
4 libSystem.B.dylib 0x00007fff89f45c08 _pthread_wqthread +
353
5 libSystem.B.dylib 0x00007fff89f45aa5 start_wqthread + 13
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007fff5fbff8b0 rcx:
0x000000010101ac00 rdx: 0x0000000000000000
rdi: 0x0000000000000000 rsi: 0x0000000100035f40 rbp:
0x00007fff5fbfec10 rsp: 0x00007fff5fbfec00
r8: 0x00007fff5fbff8b0 r9: 0x00007fff5fbff8b8 r10:
0x0000000000000001 r11: 0x00007fff5fbff0a0
r12: 0x00007fff5fbfec80 r13: 0x0000000100501810 r14:
0x000000000000000c r15: 0x0000000000000060
rip: 0x000000010006b437 rfl: 0x0000000000010246 cr2: 0x00000000000e1000
Binary Images:
0x100000000 - 0x100003ff7 +snmpd ??? (???)
<CA42A2E4-6BDE-85AC-FAAE-6FCA987193C4> /usr/sbin/snmpd
0x100008000 - 0x100034ff7 libnetsnmpagent.15.dylib 17.2.0
(compatibility 17.0.0) <98039749-650B-6F76-6325-737B36BCD895>
/usr/lib/libnetsnmpagent.15.dylib
0x100042000 - 0x10005cfff libnetsnmphelpers.15.dylib 17.2.0
(compatibility 17.0.0) <737E6394-3B05-3FED-357B-B4FF07A8658B>
/usr/lib/libnetsnmphelpers.15.dylib
0x100067000 - 0x1000f0fff libnetsnmpmibs.15.dylib 17.2.0
(compatibility 17.0.0) <7F7CD239-BE1A-0199-D2A7-2E85CF4679B5>
/usr/lib/libnetsnmpmibs.15.dylib
0x10012f000 - 0x1001a9ff7 libnetsnmp.15.dylib 17.2.0
(compatibility 17.0.0) <2BE5579E-2ADF-2930-FA8E-02E9CBB101E3>
/usr/lib/libnetsnmp.15.dylib
0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???)
<B536F2F1-9DF1-3B6C-1C2C-9075EA219A06> /usr/lib/dyld
0x7fff80139000 - 0x7fff80160ff7 libJPEG.dylib ??? (???)
<46A413EA-4FD1-A050-2EF0-6279F3EAD581>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
0x7fff801a2000 - 0x7fff804d6fef com.apple.CoreServices.CarbonCore
861.39 (861.39) <1386A24D-DD15-5903-057E-4A224FAF580B>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Car
bonCore.framework/Versions/A/CarbonCore
0x7fff804d7000 - 0x7fff80571fe7 com.apple.ApplicationServices.ATS
275.16 (???) <4B70A2FC-1902-5F27-5C3B-5C78C283C6EA>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ATS.framework/Versions/A/ATS
0x7fff80572000 - 0x7fff805acfff libcups.2.dylib 2.8.0
(compatibility 2.0.0) <7982734A-B66B-44AA-DEEC-364D2C10009B>
/usr/lib/libcups.2.dylib
0x7fff8072a000 - 0x7fff8076bfef com.apple.QD 3.36 (???)
<5DC41E81-32C9-65B2-5528-B33E934D5BB4>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/QD.framework/Versions/A/QD
0x7fff8094e000 - 0x7fff809aefe7 com.apple.framework.IOKit 2.0
(???) <4F071EF0-8260-01E9-C641-830E582FA416>
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x7fff809d3000 - 0x7fff80b91ff7 com.apple.ImageIO.framework 3.0.4
(3.0.4) <0A4F51A1-4502-767B-8A4E-F14C6214EF88>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ImageIO.framework/Versions/A/ImageIO
0x7fff80d7b000 - 0x7fff80dc3ff7 libvDSP.dylib 268.0.1
(compatibility 1.0.0) <98FC4457-F405-0262-00F7-56119CA107B6>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLi
b.framework/Versions/A/libvDSP.dylib
0x7fff80dee000 - 0x7fff81070fe7 com.apple.Foundation 6.6.7
(751.62) <6F2A5BBF-6990-D561-2928-AD61E94036D9>
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
0x7fff810e7000 - 0x7fff81370ff7 com.apple.security 6.1.2 (55002)
<4419AFFC-DAE7-873E-6A7D-5C9A5A4497A6>
/System/Library/Frameworks/Security.framework/Versions/A/Security
0x7fff81401000 - 0x7fff81c0bfe7 libBLAS.dylib 219.0.0
(compatibility 1.0.0) <FC941ECB-71D0-FAE3-DCBF-C5A619E594B8>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLi
b.framework/Versions/A/libBLAS.dylib
0x7fff81c0e000 - 0x7fff81d2dfe7 libcrypto.0.9.8.dylib 0.9.8
(compatibility 0.9.8) <14115D29-432B-CF02-6B24-A60CC533A09E>
/usr/lib/libcrypto.0.9.8.dylib
0x7fff81f60000 - 0x7fff8203dfff com.apple.vImage 4.1 (4.1)
<C3F44AA9-6F71-0684-2686-D3BBC903F020>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImag
e.framework/Versions/A/vImage
0x7fff825f6000 - 0x7fff8261efff com.apple.DictionaryServices
1.1.2 (1.1.2) <E9269069-93FA-2B71-F9BA-FDDD23C4A65E>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Dic
tionaryServices.framework/Versions/A/DictionaryServices
0x7fff8261f000 - 0x7fff82623ff7 libmathCommon.A.dylib 315.0.0
(compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5>
/usr/lib/system/libmathCommon.A.dylib
0x7fff82ad2000 - 0x7fff82af3fff libresolv.9.dylib 41.0.0
(compatibility 1.0.0) <6993F348-428F-C97E-7A84-7BD2EDC46A62>
/usr/lib/libresolv.9.dylib
0x7fff82afb000 - 0x7fff82b78fef libstdc++.6.dylib 7.9.0
(compatibility 7.0.0) <35ECA411-2C08-FD7D-11B1-1B7A04921A5C>
/usr/lib/libstdc++.6.dylib
0x7fff82eeb000 - 0x7fff82fa1ff7 libobjc.A.dylib 227.0.0
(compatibility 1.0.0) <03140531-3B2D-1EBA-DA7F-E12CC8F63969>
/usr/lib/libobjc.A.dylib
0x7fff82fa2000 - 0x7fff8369eff7 com.apple.CoreGraphics 1.545.0
(???) <58D597B1-EB3B-710E-0B8C-EC114D54E11B>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/CoreGraphics.framework/Versions/A/CoreGraphics
0x7fff836e4000 - 0x7fff836f5ff7 libz.1.dylib 1.2.3 (compatibility
1.0.0) <FB5EE53A-0534-0FFA-B2ED-486609433717> /usr/lib/libz.1.dylib
0x7fff83848000 - 0x7fff83857fff com.apple.NetFS 3.2.2 (3.2.2)
<7CCBD70E-BF31-A7A7-DB98-230687773145>
/System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
0x7fff83983000 - 0x7fff83983ff7 com.apple.Accelerate 1.6
(Accelerate 1.6) <15DF8B4A-96B2-CB4E-368D-DEC7DF6B62BB>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
0x7fff83984000 - 0x7fff83a14fff com.apple.SearchKit 1.3.0 (1.3.0)
<4175DC31-1506-228A-08FD-C704AC9DF642>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Sea
rchKit.framework/Versions/A/SearchKit
0x7fff83a15000 - 0x7fff83acefff libsqlite3.dylib 9.6.0
(compatibility 9.0.0) <2C5ED312-E646-9ADE-73A9-6199A2A43150>
/usr/lib/libsqlite3.dylib
0x7fff83af9000 - 0x7fff83af9ff7 com.apple.CoreServices 44 (44)
<DC7400FB-851E-7B8A-5BF6-6F50094302FB>
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x7fff842f1000 - 0x7fff843b2fef com.apple.ColorSync 4.6.6 (4.6.6)
<BB2C5813-C61D-3CBA-A8F7-0E59E46EBEE8>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ColorSync.framework/Versions/A/ColorSync
0x7fff84417000 - 0x7fff844d8fff libFontParser.dylib ??? (???)
<A00BB0A7-E46C-1D07-1391-194745566C7E>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ATS.framework/Versions/A/Resources/libFontParser.dylib
0x7fff845bf000 - 0x7fff8477dfff libicucore.A.dylib 40.0.0
(compatibility 1.0.0) <4274FC73-A257-3A56-4293-5968F3428854>
/usr/lib/libicucore.A.dylib
0x7fff84b4b000 - 0x7fff84b4cff7 com.apple.TrustEvaluationAgent
1.1 (1) <74800EE8-C14C-18C9-C208-20BBDB982D40>
/System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A
/TrustEvaluationAgent
0x7fff84b4d000 - 0x7fff84b8efff com.apple.SystemConfiguration
1.10.8 (1.10.2) <78D48D27-A9C4-62CA-2803-D0BBED82855A>
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemC
onfiguration
0x7fff84c88000 - 0x7fff84d9ffef libxml2.2.dylib 10.3.0
(compatibility 10.0.0) <1B27AFDD-DF87-2009-170E-C129E1572E8B>
/usr/lib/libxml2.2.dylib
0x7fff85f02000 - 0x7fff85f55ff7 com.apple.HIServices 1.8.3 (???)
<F6E0C7A7-C11D-0096-4DDA-2C77793AA6CD>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/HIServices.framework/Versions/A/HIServices
0x7fff86be5000 - 0x7fff86be7fff libRadiance.dylib ??? (???)
<A9DB4D5D-4072-971B-DEF6-DDE645F415EA>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
0x7fff86be8000 - 0x7fff86bfdff7 com.apple.LangAnalysis 1.6.6
(1.6.6) <DC999B32-BF41-94C8-0583-27D9AB463E8B>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/LangAnalysis.framework/Versions/A/LangAnalysis
0x7fff86c7a000 - 0x7fff86c7aff7 com.apple.Accelerate.vecLib 3.6
(vecLib 3.6) <4CCE5D69-F1B3-8FD3-1483-E0271DB2CCF3>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLi
b.framework/Versions/A/vecLib
0x7fff86c7b000 - 0x7fff86d2bfff edu.mit.Kerberos 6.5.11 (6.5.11)
<085D80F5-C9DC-E252-C21B-03295E660C91>
/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
0x7fff86d2c000 - 0x7fff86d7bfef libTIFF.dylib ??? (???)
<1E2593D1-A7F6-84C6-DF8F-0B46AE445926>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
0x7fff86dff000 - 0x7fff86e1cff7 libPng.dylib ??? (???)
<6D8E515B-E0A2-2BA1-9CAC-8CB8A8B35879>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ImageIO.framework/Versions/A/Resources/libPng.dylib
0x7fff87299000 - 0x7fff872c4ff7 libxslt.1.dylib 3.24.0
(compatibility 3.0.0) <8AB4CA9E-435A-33DA-7041-904BA7FA11D5>
/usr/lib/libxslt.1.dylib
0x7fff872c5000 - 0x7fff87365fff com.apple.LaunchServices 362.3
(362.3) <B90B7C31-FEF8-3C26-BFB3-D8A48BD2C0DA>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Lau
nchServices.framework/Versions/A/LaunchServices
0x7fff87597000 - 0x7fff875d2fff com.apple.AE 496.5 (496.5)
<208DF391-4DE6-81ED-C697-14A2930D1BC6>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.
framework/Versions/A/AE
0x7fff875d3000 - 0x7fff875d9ff7 com.apple.DiskArbitration 2.3
(2.3) <857F6E43-1EF4-7D53-351B-10DE0A8F992A>
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitra
tion
0x7fff8795f000 - 0x7fff879a9ff7 com.apple.Metadata 10.6.3
(507.15) <2EF19055-D7AE-4D77-E589-7B71B0BC1E59>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Met
adata.framework/Versions/A/Metadata
0x7fff87a80000 - 0x7fff87a96fef libbsm.0.dylib ??? (???)
<42D3023A-A1F7-4121-6417-FCC6B51B3E90> /usr/lib/libbsm.0.dylib
0x7fff87a97000 - 0x7fff87b1cff7
com.apple.print.framework.PrintCore 6.3 (312.7) <CDFE82DD-D811-A091-179F-6E76069B432D>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/PrintCore.framework/Versions/A/PrintCore
0x7fff87b1d000 - 0x7fff87b87fe7 libvMisc.dylib 268.0.1
(compatibility 1.0.0) <514D400C-50A5-C196-83AA-1035DDC8FBBE>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLi
b.framework/Versions/A/libvMisc.dylib
0x7fff87c82000 - 0x7fff87c82ff7 com.apple.ApplicationServices 38
(38) <10A0B9E9-4988-03D4-FC56-DDE231A02C63>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Applica
tionServices
0x7fff87c97000 - 0x7fff87c9cfff libGIF.dylib ??? (???)
<201B8077-B5CC-11AA-E1B0-1D057ABE416A>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
0x7fff89023000 - 0x7fff89031ff7 libkxld.dylib ??? (???)
<8145A534-95CC-9F3C-B78B-AC9898F38C6F> /usr/lib/system/libkxld.dylib
0x7fff89032000 - 0x7fff891a9fe7 com.apple.CoreFoundation 6.6.5
(550.43) <31A1C118-AD96-0A11-8BDF-BD55B9940EDC>
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundati
on
0x7fff892a3000 - 0x7fff892b7ff7
com.apple.speech.synthesis.framework 3.10.35 (3.10.35) <621B7415-A0B9-07A7-F313-36BEEDD7B132>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
0x7fff89378000 - 0x7fff897bbfef libLAPACK.dylib 219.0.0
(compatibility 1.0.0) <0CC61C98-FF51-67B3-F3D8-C5E430C201A9>
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLi
b.framework/Versions/A/libLAPACK.dylib
0x7fff89ceb000 - 0x7fff89dbffe7 com.apple.CFNetwork 454.12.4
(454.12.4) <C83E2BA1-1818-B3E8-5334-860AD21D1C80>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFN
etwork.framework/Versions/A/CFNetwork
0x7fff89f2b000 - 0x7fff8a0ecfef libSystem.B.dylib 125.2.11
(compatibility 1.0.0) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69>
/usr/lib/libSystem.B.dylib
0x7fff8a0ed000 - 0x7fff8a1aafff com.apple.CoreServices.OSServices
359.2 (359.2) <BBB8888E-18DE-5D09-3C3A-F4C029EC7886>
/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSS
ervices.framework/Versions/A/OSServices
0x7fff8a4f0000 - 0x7fff8a53cfff libauto.dylib ??? (???)
<F7221B46-DC4F-3153-CE61-7F52C8C293CF> /usr/lib/libauto.dylib
0x7fff8a5c7000 - 0x7fff8a645ff7 com.apple.CoreText 151.10 (???)
<54961997-55D8-DC0F-2634-674E452D5A8E>
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Framewo
rks/CoreText.framework/Versions/A/CoreText
0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???)
<9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib