hwmond is causing up to 70% cpu load after every reboot on several xserves i manage caused by the update to 10.6.8. anybody else having this problem? it can be solved by hwmond -s5 to set the hwmond interval to 5 seconds but this command is not processed properly, is causing an error message "IOConnectMethodScalarIScalra0 call return err = 0xE00002BC" and has to be exited with cmd+c. Also, the com.apple.hwmond.plist in LaunchDaemons is empty.
Can someone please post a sample of the hwmond process when it's spinning hard? Select the process in Activity Monitor, then click on the Inspect button in the toolbar, then click on the Sample button at the bottom of the window. That should help with diagnosing what happens.
--Paul
Analysis of sampling hwmond (pid 111) every 1 millisecond
Call graph:
553 Thread_435 DispatchQueue_1: com.apple.main-thread (serial)
553 0x20d9
553 0x276d
553 CFRunLoopRun
553 CFRunLoopRunSpecific
553 __CFRunLoopRun
551 __CFMachPortPerform
551 IODispatchCalloutFromCFMessage
475 0x99ac
318 0x9658
318 IOUSBDeviceClass::SetConfiguration(unsigned char)
318 IOConnectCallScalarMethod
318 IOConnectCallMethod
318 io_connect_method
318 mach_msg
318 mach_msg_trap
123 0x96cf
122 0x9258
122 IOCreatePlugInInterfaceForService
122 IOUSBInterfaceClass::start(__CFDictionary const*, unsigned int)
122 IOUSBInterfaceClass::GetPropertyInfo()
118 IORegistryEntryCreateCFProperties
73 io_registry_entry_get_properties
73 mach_msg
73 mach_msg_trap
43 IOCFUnserialize
41 IOCFUnserializeparse
16 buildString
13 CFStringCreateWithCString
9 __CFStringCreateImmutableFunnel3
8 __CFStringCreateImmutableFunnel3
1 dyld_stub_malloc_zone_malloc
2 strlen
1 CFStringCreateWithCString
1 strncmp
2 free
1 spin_unlock
1 szone_free_definite_size
1 CFDictionarySetValue
1 CFBasicHashSetValue
1 __CFBasicHashAddValue
1 __CFBasicHashRehash
13 CFDictionarySetValue
13 CFBasicHashSetValue
12 __CFBasicHashAddValue
7 __CFBasicHashRehash
3 malloc_zone_malloc
1 __spin_lock
1 dyld_stub__spin_lock
1 szone_malloc_should_clear
1 spin_lock
2 szone_free
1 szone_free
1 tiny_free_list_add_ptr
1 CFGetAllocator
1 malloc_zone_free
1 OSAtomicCompareAndSwapPtrBarrier
1 __compare_and_swap32
1 __CFBasicHashFindNewMarker
1 __CFDictionaryCallback
1 CFGetAllocator
1 ___CFBasicHashFindBucket1
1 __CFBasicHashStandardCallback
1 __compare_and_swap32
1 ___CFBasicHashFindBucket1
3 newObject
3 malloc
3 malloc_zone_malloc
3 szone_malloc_should_clear
2 szone_malloc_should_clear
1 tiny_malloc_from_free_list
2 IOCFUnserializeparse
2 getString
1 CFNumberCreate
1 _CFRuntimeCreateInstance
1 malloc_zone_malloc
1 szone_malloc_should_clear
1 OSAtomicCompareAndSwapPtrBarrier
1 __compare_and_swap32
1 getTag
1 strtol
1 strtoul
1 CFDictionaryCreateMutable
1 __CFDictionaryCreateGeneric
1 CFBasicHashCreate
1 dyld_stub_pthread_getspecific
1 szone_free_definite_size
2 vm_deallocate
2 mach_msg
2 mach_msg_trap
3 _CFRelease
3 __CFBasicHashDrain
3 __CFBasicHashStandardCallback
2 _CFRelease
1 __CFBasicHashDrain
1 szone_free
1 OSAtomicCompareAndSwapPtrBarrier
1 __compare_and_swap32
1 CFDictionaryGetValue
1 CFBasicHashFindBucket
1 ___CFBasicHashFindBucket1
1 __CFStringHash
1 0x938d
1 0x8f46
1 IOUSBInterfaceClass::GetPipeProperties(unsigned char, unsigned char*, unsigned char*, unsigned char*, unsigned short*, unsigned char*)
1 IOConnectCallScalarMethod
1 IOConnectCallMethod
1 io_connect_method
1 mach_msg
1 mach_msg_trap
30 0x94de
30 IOCreatePlugInInterfaceForService
28 IOUSBDeviceClass::start(__CFDictionary const*, unsigned int)
23 IORegistryEntryCreateCFProperties
14 IOCFUnserialize
13 IOCFUnserializeparse
4 CFDictionarySetValue
4 CFBasicHashSetValue
3 __CFBasicHashAddValue
1 OSAtomicCompareAndSwapPtrBarrier
1 __compare_and_swap32
1 _CFRetain
1 OSAtomicCompareAndSwapPtrBarrier
1 __CFBasicHashRehash
1 ___CFBasicHashFindBucket1
1 ___CFBasicHashFindBucket1
1 __CFBasicHashStandardCallback
4 buildString
3 CFStringCreateWithCString
2 __CFStringCreateImmutableFunnel3
1 _CFRuntimeCreateInstance
1 __CFAllocatorSystemAllocate
1 __memcpy
1 strlen
1 CFDictionarySetValue
1 CFBasicHashSetValue
1 __CFBasicHashAddValue
1 OSAtomicCompareAndSwapPtrBarrier
1 __compare_and_swap32
2 getString
1 CFNumberCreate
1 IOCFUnserializeparse
1 getTag
1 free
1 szone_free_definite_size
5 io_registry_entry_get_properties
5 mach_msg
5 mach_msg_trap
4 vm_deallocate
4 mach_msg
4 mach_msg_trap
2 IOUSBDeviceClass::CacheConfigDescriptor()
2 IOConnectCallMethod
2 io_connect_method
2 mach_msg
2 mach_msg_trap
1 IOServiceOpen
1 io_service_open_extended
1 mach_msg
1 mach_msg_trap
1 IOUSBDeviceClass::start(__CFDictionary const*, unsigned int)
1 _CFRelease
1 __CFBasicHashDrain
1 __CFBasicHashStandardCallback
1 _CFRelease
1 szone_free
1 CFPlugInCreate
1 _CFBundleCreate
1 _CFBundleCopyBundleForURL
1 -[__CFx606449CMT objectForKey:]
1 probeGC
1 objectEqual
1 CFEqual
1 IORegistryEntryCreateCFProperty
1 io_registry_entry_get_property
1 mach_msg
1 mach_msg_trap
3 0x95c0
3 IOUSBDeviceClass::queryInterface(CFUUIDBytes, void**)
2 IOObjectRelease
2 mach_port_deallocate
1 mach_msg
1 mach_msg_trap
1 mig_get_reply_port
1 __CFUUIDCreateWithBytesPrimitive
1 CFDictionaryGetValue
1 CFBasicHashFindBucket
1 ___CFBasicHashFindBucket1
1 0x95eb
1 IOUSBDeviceClass::deviceUSBDeviceOpen(void*)
1 IOUSBDeviceClass::USBDeviceOpen(bool)
1 IOConnectCallScalarMethod
1 IOConnectCallMethod
1 io_connect_method
1 mach_msg
1 mach_msg_trap
47 0x998a
47 pthread_mutex_lock
47 semaphore_wait_signal_trap
25 0x9972
10 0x8fab
10 IOUSBIUnknown::release()
9 IOUSBInterfaceClass::~IOUSBInterfaceClass()
8 IOServiceClose
7 io_service_close
7 mach_msg
7 mach_msg_trap
1 mach_port_deallocate
1 mach_msg
1 mach_msg_trap
1 IOObjectRelease
1 mach_port_deallocate
1 mach_msg
1 mach_msg_trap
1 free
1 szone_size
8 0x8fc7
8 IOUSBDeviceClass::USBDeviceClose()
8 IOConnectCallScalarMethod
8 IOConnectCallMethod
8 io_connect_method
8 mach_msg
8 mach_msg_trap
6 0x8fd4
6 IOUSBIUnknown::release()
6 IOUSBDeviceClass::~IOUSBDeviceClass()
4 IOServiceClose
4 io_service_close
4 mach_msg
4 mach_msg_trap
2 IOObjectRelease
2 mach_port_deallocate
2 mach_msg
2 mach_msg_trap
1 0x8f9e
1 IOUSBInterfaceClass::USBInterfaceClose()
1 IOConnectCallScalarMethod
1 IOConnectCallMethod
1 io_connect_method
1 mach_msg
1 mach_msg_trap
2 mach_port_deallocate
1 mach_msg
1 mach_msg_trap
1 mig_get_reply_port
2 mach_port_get_refs
2 mach_msg
2 mach_msg_trap
2 mach_port_insert_member
2 mach_msg
2 mach_msg_trap
553 Thread_816 DispatchQueue_2: com.apple.libdispatch-manager (serial)
553 start_wqthread
553 _pthread_wqthread
553 _dispatch_worker_thread2
553 _dispatch_queue_invoke
553 _dispatch_mgr_invoke
553 kevent
553 Thread_863
553 thread_start
553 _pthread_start
552 0x9ce2
552 sleep$UNIX2003
552 nanosleep$UNIX2003
552 __semwait_signal
1 0x9cd6
1 vm_deallocate
1 mach_msg
1 mach_msg_trap
553 Thread_864
553 thread_start
553 _pthread_start
310 0x9a14
310 sleep$UNIX2003
310 nanosleep$UNIX2003
310 __semwait_signal
150 0x9a77
150 usleep$UNIX2003
150 nanosleep$UNIX2003
150 __semwait_signal
93 0x9a04
47 0x90bd
47 IOUSBInterfaceClass::interfaceWritePipe(void*, unsigned char, void*, unsigned long)
47 IOUSBInterfaceClass::WritePipe(unsigned char, void*, unsigned long, unsigned long, unsigned long)
47 IOConnectCallMethod
47 io_connect_method
46 mach_msg
46 mach_msg_trap
1 io_connect_method
46 0x907e
46 pthread_mutex_lock
46 semaphore_wait_signal_trap
Total number in stack (recursive counted multiple, when >=5):
23 mach_msg
23 mach_msg_trap
7 IOConnectCallMethod
7 io_connect_method
6 OSAtomicCompareAndSwapPtrBarrier
6 ___CFBasicHashFindBucket1
6 __compare_and_swap32
5 IOConnectCallScalarMethod
5 mach_port_deallocate
Sort by top of stack, same collapsed (when >= 5):
__semwait_signal 1012
kevent 553
mach_msg_trap 485
semaphore_wait_signal_trap 93
__CFStringCreateImmutableFunnel3 8
__compare_and_swap32 6
Thanks. The interesting part is here:
551 __CFMachPortPerform
551 IODispatchCalloutFromCFMessage
475 0x99ac
318 0x9658
318 IOUSBDeviceClass::SetConfiguration(unsigned char)
318 IOConnectCallScalarMethod
318 IOConnectCallMethod
318 io_connect_method
318 mach_msg
318 mach_msg_trap
123 0x96cf
122 0x9258
122 IOCreatePlugInInterfaceForService
122 IOUSBInterfaceClass::start(__CFDictionary const*, unsigned int)
122 IOUSBInterfaceClass::GetPropertyInfo()
118 IORegistryEntryCreateCFProperties
73 io_registry_entry_get_properties
73 mach_msg
73 mach_msg_trap
The process is spending an awful lot of time in IOUSBDeviceClass::SetConfiguration and IOUSBInterfaceClass::start. The other function calls with high counts are pretty much expected calls. What USB devices do people have attached to their Xserves? UPS units, keyboards, mice, KVMs, external disk drives?
--Paul
I have no USB devices connected to either server. They are both headless in a data center.
The above was from a headless Xserve, no peripherals.
Btw, I tried the above "solution" and zapped pram a few times -- absolutely did nothing.
Ours is headless as well, however we have a USB drive attached.
Both of my servers are headless production servers in a data center, no peripherals attached.
Thanks for the input.
Running 'sudo lsof | grep hwmond' on an Xserve gives the following output:
hwmond 73 root cwd DIR 14,2 1156 2 /
hwmond 73 root txt REG 14,2 87040 446400 /usr/sbin/hwmond
hwmond 73 root txt REG 14,2 586688 448475 /System/Library/PrivateFrameworks/CoreRAID.framework/Versions/A/CoreRAID
hwmond 73 root txt REG 14,2 364464 453954 /System/Library/PrivateFrameworks/PlatformHardwareManagement.framework/Versions /A/PlatformHardwareManagement
hwmond 73 root txt REG 14,2 235568 711329 /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBLib.bundle/Co ntents/MacOS/IOUSBLib
hwmond 73 root txt REG 14,2 177312 464611 /System/Library/Extensions/IOSCSIArchitectureModelFamily.kext/Contents/PlugIns/ SCSITaskUserClient.kext/Contents/PlugIns/SCSITaskLib.plugin/Contents/MacOS/SCSIT askLib
hwmond 73 root txt REG 14,2 1054960 497707 /usr/lib/dyld
hwmond 73 root txt REG 14,2 207499264 756991 /private/var/db/dyld/dyld_shared_cache_i386
hwmond 73 root 0r CHR 3,2 0t0 297 /dev/null
hwmond 73 root 1 PIPE 0xffffff801387d6b0 16384 ->0xffffff801387d760
hwmond 73 root 2 PIPE 0xffffff801387d6b0 16384 ->0xffffff801387d760
hwmond 73 root 3u KQUEUE count=1, state=0x2
The interesting line is that hwmond is using IOUSBLib, which a quick check of the .bom file for the 10.6.8 update shows as being one of the changed elements. So, we can try swapping out the 10.6.7 IOUSBLib instead to see if this helps.
DON'T EVEN THINK ABOUT TRYING THIS IF YOU'RE NOT THOROUGHLY COMFORTABLE ON THE COMMAND LINE OR IF YOU'RE WORKING ON A PRODUCTION SERVER!!!!!!!! YOU SHOULD BE AT THE SERVER CONSOLE AND HAVE A MONITOR, KEYBOARD, AND MOUSE AVAILABLE. THIS HAS THE POTENTIAL TO CAUSE SERIOUS PROBLEMS WITH YOUR SERVER IF YOU CAN'T HANDLE THE RECOVERY PROCEDURE.
1) Make a copy of the IOUSBLib.bundle to swap it back in later if need be:
cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns
tar zcf /Users/Shared/IOUSBLib.bundle.10.6.8.tgz IOUSBLib.bundle
2) On a Mac (doesn't need to be a server) that is running 10.6.7, get a copy of the IOUSBLib.bundle:
cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns
tar zcf /Users/Shared/IOUSBLib.bundle.10.6.7.tgz IOUSBLib.bundle
3) Copy the IOUSBLib.bundle.10.6.7.tgz file over the the Xserve running 10.6.8 into the /Users/Shared directory.
4) Move the existing bundle aside:
cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns
sudo mv IOUSBLib.bundle /System/Library/IOUSB.bundle
DO NOT DELETE THE BUNDLE!
5) Drop the 10.6.7 version into place:
cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns
sudo tar zxf /Users/Shared/IOUSBLib.bundle.10.6.7.tgz
6) Reboot in safe mode (shift key held down) to force a rebuild of the kext cache with fingers crossed.
7) Check to see if hwmond is actually running and is behaving properly. If so, reboot again (non-safe mode) to see if it works with all extensions enabled.
Recovery Procedure If Needed
1) Boot Xserve into target disk mode, plug into another Mac via FireWire.
2) Delete the 10.6.7 IOUSBLib.bundle from /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns.
sudo rm -rf /Volumes/<server HD name>/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBLib.bund le
3) Move the 10.6.8 IOUSBLib.bundle back into place
sudo mv /Volumes/<server HD name>/System/Library/IOUSBLib.bundle /Volumes/<server HD name>/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBLib.bund le
4) Reboot in safe mode (shift key held down) to force a rebuild of the kext cache with fingers crossed.
5) Check to see if hwmond is actually running and is behaving properly. If so, reboot again (non-safe mode) to see if it works with all extensions enabled.
It's late here on the East coast, so I'll wrap it up for tonight.
--Paul
This is strange... Having zapped the PRAM and had HWMOND working as normal, upon restarting the server it's gone back to 85% CPU.. I'm going to play around and see what I can do.. If anyone has any suggestions they'd like to try then let me know and i'll try them on my server here.
Good Luck
Beatle
Awesome find, thank you Paul. I'll give this a shot today. FWIW, mine's headless as well, though yesterday I connected a VGA cable & monitor. It has an APC SMX1500RM2U UPS connected via USB and there are two Apple, dual-port, 1Gbps NICs installed in each PCIe slot. Beyond that it has 10GB of RAM, and 3 stock 2TB ADM's, Hitachi, model # HDS722020ALA330. Not all of the RAM is from Apple/Hynix, etc., but at this point it doesn't matter.
I tried resetting the NVRAM as well, no luck, same results. System Profiler shows the front LED/control panel being connected via USB (I think), so I wonder if this is related:
Frontpanel Controller:
Product ID: 0x8261
Vendor ID: 0x05ac (Apple Inc.)
Version: 0.10
Speed: Up to 12 Mb/sec
Manufacturer: Apple Computer
Location ID: 0x3d100000 / 2
Current Available (mA): 500
Current Required (mA): Unknown (Device has not been configured)
System Profiler shows the front LED/control panel being connected via USB (I think), so I wonder if this is related:
Good catch!
--Paul
Exactly the same thing here, 5 intel xserves, all of them have HWMOND using 87% of one core.
A friend has two Mac Mini servers and they don't suffer from this issue. Im thinking its the LED panel on the front of the Xserve.
More specifically I think it's the interaction between hwmond, the front panel, and, from what paulsuh found, possibly the IOUSB.bundle in his earlier post as of 10.6.8. This might explain the choppy processor LED behavior I thought I saw a few days back. Feels like we're making more progress on this amongst ourselves than Apple is (zing!).
Probably because we have xServes and Apple doesn't ;-)
We have 5 Xserves and 2 MacMini servers here and I can confirm this behavior. The Minis are fine, but the Xserves are using almost an entire processor core.
hwmond high cpu load after 10.6.8 update
