hwmond high cpu load after 10.6.8 update

Can someone please post a sample of the hwmond process when it's spinning hard? Select the process in Activity Monitor, then click on the Inspect button in the toolbar, then click on the Sample button at the bottom of the window. That should help with diagnosing what happens.

--Paul

Analysis of sampling hwmond (pid 111) every 1 millisecond

Call graph:

553 Thread_435 DispatchQueue_1: com.apple.main-thread (serial)

553 0x20d9

553 0x276d

553 CFRunLoopRun

553 CFRunLoopRunSpecific

553 __CFRunLoopRun

551 __CFMachPortPerform

551 IODispatchCalloutFromCFMessage

475 0x99ac

318 0x9658

318 IOUSBDeviceClass::SetConfiguration(unsigned char)

318 IOConnectCallScalarMethod

318 IOConnectCallMethod

318 io_connect_method

318 mach_msg

318 mach_msg_trap

123 0x96cf

122 0x9258

122 IOCreatePlugInInterfaceForService

122 IOUSBInterfaceClass::start(__CFDictionary const*, unsigned int)

122 IOUSBInterfaceClass::GetPropertyInfo()

118 IORegistryEntryCreateCFProperties

73 io_registry_entry_get_properties

73 mach_msg

73 mach_msg_trap

43 IOCFUnserialize

41 IOCFUnserializeparse

16 buildString

13 CFStringCreateWithCString

9 __CFStringCreateImmutableFunnel3

8 __CFStringCreateImmutableFunnel3

1 dyld_stub_malloc_zone_malloc

2 strlen

1 CFStringCreateWithCString

1 strncmp

2 free

1 spin_unlock

1 szone_free_definite_size

1 CFDictionarySetValue

1 CFBasicHashSetValue

1 __CFBasicHashAddValue

1 __CFBasicHashRehash

13 CFDictionarySetValue

13 CFBasicHashSetValue

12 __CFBasicHashAddValue

7 __CFBasicHashRehash

3 malloc_zone_malloc

1 __spin_lock

1 dyld_stub__spin_lock

1 szone_malloc_should_clear

1 spin_lock

2 szone_free

1 szone_free

1 tiny_free_list_add_ptr

1 CFGetAllocator

1 malloc_zone_free

1 OSAtomicCompareAndSwapPtrBarrier

1 __compare_and_swap32

1 __CFBasicHashFindNewMarker

1 __CFDictionaryCallback

1 CFGetAllocator

1 ___CFBasicHashFindBucket1

1 __CFBasicHashStandardCallback

1 __compare_and_swap32

1 ___CFBasicHashFindBucket1

3 newObject

3 malloc

3 malloc_zone_malloc

3 szone_malloc_should_clear

2 szone_malloc_should_clear

1 tiny_malloc_from_free_list

2 IOCFUnserializeparse

2 getString

1 CFNumberCreate

1 _CFRuntimeCreateInstance

1 malloc_zone_malloc

1 szone_malloc_should_clear

1 OSAtomicCompareAndSwapPtrBarrier

1 __compare_and_swap32

1 getTag

1 strtol

1 strtoul

1 CFDictionaryCreateMutable

1 __CFDictionaryCreateGeneric

1 CFBasicHashCreate

1 dyld_stub_pthread_getspecific

1 szone_free_definite_size

2 vm_deallocate

2 mach_msg

2 mach_msg_trap

3 _CFRelease

3 __CFBasicHashDrain

3 __CFBasicHashStandardCallback

2 _CFRelease

1 __CFBasicHashDrain

1 szone_free

1 OSAtomicCompareAndSwapPtrBarrier

1 __compare_and_swap32

1 CFDictionaryGetValue

1 CFBasicHashFindBucket

1 ___CFBasicHashFindBucket1

1 __CFStringHash

1 0x938d

1 0x8f46

1 IOUSBInterfaceClass::GetPipeProperties(unsigned char, unsigned char*, unsigned char*, unsigned char*, unsigned short*, unsigned char*)

1 IOConnectCallScalarMethod

1 IOConnectCallMethod

1 io_connect_method

1 mach_msg

1 mach_msg_trap

30 0x94de

30 IOCreatePlugInInterfaceForService

28 IOUSBDeviceClass::start(__CFDictionary const*, unsigned int)

23 IORegistryEntryCreateCFProperties

14 IOCFUnserialize

13 IOCFUnserializeparse

4 CFDictionarySetValue

4 CFBasicHashSetValue

3 __CFBasicHashAddValue

1 OSAtomicCompareAndSwapPtrBarrier

1 __compare_and_swap32

1 _CFRetain

1 OSAtomicCompareAndSwapPtrBarrier

1 __CFBasicHashRehash

1 ___CFBasicHashFindBucket1

1 ___CFBasicHashFindBucket1

1 __CFBasicHashStandardCallback

4 buildString

3 CFStringCreateWithCString

2 __CFStringCreateImmutableFunnel3

1 _CFRuntimeCreateInstance

1 __CFAllocatorSystemAllocate

1 __memcpy

1 strlen

1 CFDictionarySetValue

1 CFBasicHashSetValue

1 __CFBasicHashAddValue

1 OSAtomicCompareAndSwapPtrBarrier

1 __compare_and_swap32

2 getString

1 CFNumberCreate

1 IOCFUnserializeparse

1 getTag

1 free

1 szone_free_definite_size

5 io_registry_entry_get_properties

5 mach_msg

5 mach_msg_trap

4 vm_deallocate

4 mach_msg

4 mach_msg_trap

2 IOUSBDeviceClass::CacheConfigDescriptor()

2 IOConnectCallMethod

2 io_connect_method

2 mach_msg

2 mach_msg_trap

1 IOServiceOpen

1 io_service_open_extended

1 mach_msg

1 mach_msg_trap

1 IOUSBDeviceClass::start(__CFDictionary const*, unsigned int)

1 _CFRelease

1 __CFBasicHashDrain

1 __CFBasicHashStandardCallback

1 _CFRelease

1 szone_free

1 CFPlugInCreate

1 _CFBundleCreate

1 _CFBundleCopyBundleForURL

1 -[__CFx606449CMT objectForKey:]

1 probeGC

1 objectEqual

1 CFEqual

1 IORegistryEntryCreateCFProperty

1 io_registry_entry_get_property

1 mach_msg

1 mach_msg_trap

3 0x95c0

3 IOUSBDeviceClass::queryInterface(CFUUIDBytes, void**)

2 IOObjectRelease

2 mach_port_deallocate

1 mach_msg

1 mach_msg_trap

1 mig_get_reply_port

1 __CFUUIDCreateWithBytesPrimitive

1 CFDictionaryGetValue

1 CFBasicHashFindBucket

1 ___CFBasicHashFindBucket1

1 0x95eb

1 IOUSBDeviceClass::deviceUSBDeviceOpen(void*)

1 IOUSBDeviceClass::USBDeviceOpen(bool)

1 IOConnectCallScalarMethod

1 IOConnectCallMethod

1 io_connect_method

1 mach_msg

1 mach_msg_trap

47 0x998a

47 pthread_mutex_lock

47 semaphore_wait_signal_trap

25 0x9972

10 0x8fab

10 IOUSBIUnknown::release()

9 IOUSBInterfaceClass::~IOUSBInterfaceClass()

8 IOServiceClose

7 io_service_close

7 mach_msg

7 mach_msg_trap

1 mach_port_deallocate

1 mach_msg

1 mach_msg_trap

1 IOObjectRelease

1 mach_port_deallocate

1 mach_msg

1 mach_msg_trap

1 free

1 szone_size

8 0x8fc7

8 IOUSBDeviceClass::USBDeviceClose()

8 IOConnectCallScalarMethod

8 IOConnectCallMethod

8 io_connect_method

8 mach_msg

8 mach_msg_trap

6 0x8fd4

6 IOUSBIUnknown::release()

6 IOUSBDeviceClass::~IOUSBDeviceClass()

4 IOServiceClose

4 io_service_close

4 mach_msg

4 mach_msg_trap

2 IOObjectRelease

2 mach_port_deallocate

2 mach_msg

2 mach_msg_trap

1 0x8f9e

1 IOUSBInterfaceClass::USBInterfaceClose()

1 IOConnectCallScalarMethod

1 IOConnectCallMethod

1 io_connect_method

1 mach_msg

1 mach_msg_trap

2 mach_port_deallocate

1 mach_msg

1 mach_msg_trap

1 mig_get_reply_port

2 mach_port_get_refs

2 mach_msg

2 mach_msg_trap

2 mach_port_insert_member

2 mach_msg

2 mach_msg_trap

553 Thread_816 DispatchQueue_2: com.apple.libdispatch-manager (serial)

553 start_wqthread

553 _pthread_wqthread

553 _dispatch_worker_thread2

553 _dispatch_queue_invoke

553 _dispatch_mgr_invoke

553 kevent

553 Thread_863

553 thread_start

553 _pthread_start

552 0x9ce2

552 sleep$UNIX2003

552 nanosleep$UNIX2003

552 __semwait_signal

1 0x9cd6

1 vm_deallocate

1 mach_msg

1 mach_msg_trap

553 Thread_864

553 thread_start

553 _pthread_start

310 0x9a14

310 sleep$UNIX2003

310 nanosleep$UNIX2003

310 __semwait_signal

150 0x9a77

150 usleep$UNIX2003

150 nanosleep$UNIX2003

150 __semwait_signal

93 0x9a04

47 0x90bd

47 IOUSBInterfaceClass::interfaceWritePipe(void*, unsigned char, void*, unsigned long)

47 IOUSBInterfaceClass::WritePipe(unsigned char, void*, unsigned long, unsigned long, unsigned long)

47 IOConnectCallMethod

47 io_connect_method

46 mach_msg

46 mach_msg_trap

1 io_connect_method

46 0x907e

46 pthread_mutex_lock

46 semaphore_wait_signal_trap

Total number in stack (recursive counted multiple, when >=5):

23 mach_msg

23 mach_msg_trap

7 IOConnectCallMethod

7 io_connect_method

6 OSAtomicCompareAndSwapPtrBarrier

6 ___CFBasicHashFindBucket1

6 __compare_and_swap32

5 IOConnectCallScalarMethod

5 mach_port_deallocate

Sort by top of stack, same collapsed (when >= 5):

__semwait_signal 1012

kevent 553

mach_msg_trap 485

semaphore_wait_signal_trap 93

__CFStringCreateImmutableFunnel3 8

__compare_and_swap32 6

Thanks. The interesting part is here:

551 __CFMachPortPerform

551 IODispatchCalloutFromCFMessage

475 0x99ac

318 0x9658

318 IOUSBDeviceClass::SetConfiguration(unsigned char)

318 IOConnectCallScalarMethod

318 IOConnectCallMethod

318 io_connect_method

318 mach_msg

318 mach_msg_trap

123 0x96cf

122 0x9258

122 IOCreatePlugInInterfaceForService

122 IOUSBInterfaceClass::start(__CFDictionary const*, unsigned int)

122 IOUSBInterfaceClass::GetPropertyInfo()

118 IORegistryEntryCreateCFProperties

73 io_registry_entry_get_properties

73 mach_msg

73 mach_msg_trap

The process is spending an awful lot of time in IOUSBDeviceClass::SetConfiguration and IOUSBInterfaceClass::start. The other function calls with high counts are pretty much expected calls. What USB devices do people have attached to their Xserves? UPS units, keyboards, mice, KVMs, external disk drives?

--Paul

I have no USB devices connected to either server. They are both headless in a data center.

The above was from a headless Xserve, no peripherals.

Btw, I tried the above "solution" and zapped pram a few times -- absolutely did nothing.

Ours is headless as well, however we have a USB drive attached.

Both of my servers are headless production servers in a data center, no peripherals attached.

Thanks for the input.

Running 'sudo lsof | grep hwmond' on an Xserve gives the following output:

hwmond 73 root cwd DIR 14,2 1156 2 /

hwmond 73 root txt REG 14,2 87040 446400 /usr/sbin/hwmond

hwmond 73 root txt REG 14,2 586688 448475 /System/Library/PrivateFrameworks/CoreRAID.framework/Versions/A/CoreRAID

hwmond 73 root txt REG 14,2 364464 453954 /System/Library/PrivateFrameworks/PlatformHardwareManagement.framework/Versions /A/PlatformHardwareManagement

hwmond 73 root txt REG 14,2 235568 711329 /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBLib.bundle/Co ntents/MacOS/IOUSBLib

hwmond 73 root txt REG 14,2 177312 464611 /System/Library/Extensions/IOSCSIArchitectureModelFamily.kext/Contents/PlugIns/ SCSITaskUserClient.kext/Contents/PlugIns/SCSITaskLib.plugin/Contents/MacOS/SCSIT askLib

hwmond 73 root txt REG 14,2 1054960 497707 /usr/lib/dyld

hwmond 73 root txt REG 14,2 207499264 756991 /private/var/db/dyld/dyld_shared_cache_i386

hwmond 73 root 0r CHR 3,2 0t0 297 /dev/null

hwmond 73 root 1 PIPE 0xffffff801387d6b0 16384 ->0xffffff801387d760

hwmond 73 root 2 PIPE 0xffffff801387d6b0 16384 ->0xffffff801387d760

hwmond 73 root 3u KQUEUE count=1, state=0x2

The interesting line is that hwmond is using IOUSBLib, which a quick check of the .bom file for the 10.6.8 update shows as being one of the changed elements. So, we can try swapping out the 10.6.7 IOUSBLib instead to see if this helps.

DON'T EVEN THINK ABOUT TRYING THIS IF YOU'RE NOT THOROUGHLY COMFORTABLE ON THE COMMAND LINE OR IF YOU'RE WORKING ON A PRODUCTION SERVER!!!!!!!! YOU SHOULD BE AT THE SERVER CONSOLE AND HAVE A MONITOR, KEYBOARD, AND MOUSE AVAILABLE. THIS HAS THE POTENTIAL TO CAUSE SERIOUS PROBLEMS WITH YOUR SERVER IF YOU CAN'T HANDLE THE RECOVERY PROCEDURE.

1) Make a copy of the IOUSBLib.bundle to swap it back in later if need be:

cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns

tar zcf /Users/Shared/IOUSBLib.bundle.10.6.8.tgz IOUSBLib.bundle

2) On a Mac (doesn't need to be a server) that is running 10.6.7, get a copy of the IOUSBLib.bundle:

cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns

tar zcf /Users/Shared/IOUSBLib.bundle.10.6.7.tgz IOUSBLib.bundle

3) Copy the IOUSBLib.bundle.10.6.7.tgz file over the the Xserve running 10.6.8 into the /Users/Shared directory.

4) Move the existing bundle aside:

cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns

sudo mv IOUSBLib.bundle /System/Library/IOUSB.bundle

DO NOT DELETE THE BUNDLE!

5) Drop the 10.6.7 version into place:

cd /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns

sudo tar zxf /Users/Shared/IOUSBLib.bundle.10.6.7.tgz

6) Reboot in safe mode (shift key held down) to force a rebuild of the kext cache with fingers crossed.

7) Check to see if hwmond is actually running and is behaving properly. If so, reboot again (non-safe mode) to see if it works with all extensions enabled.

Recovery Procedure If Needed

1) Boot Xserve into target disk mode, plug into another Mac via FireWire.

2) Delete the 10.6.7 IOUSBLib.bundle from /System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns.

sudo rm -rf /Volumes/<server HD name>/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBLib.bund le

3) Move the 10.6.8 IOUSBLib.bundle back into place

sudo mv /Volumes/<server HD name>/System/Library/IOUSBLib.bundle /Volumes/<server HD name>/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/IOUSBLib.bund le

4) Reboot in safe mode (shift key held down) to force a rebuild of the kext cache with fingers crossed.

5) Check to see if hwmond is actually running and is behaving properly. If so, reboot again (non-safe mode) to see if it works with all extensions enabled.

It's late here on the East coast, so I'll wrap it up for tonight.

--Paul

This is strange... Having zapped the PRAM and had HWMOND working as normal, upon restarting the server it's gone back to 85% CPU.. I'm going to play around and see what I can do.. If anyone has any suggestions they'd like to try then let me know and i'll try them on my server here.

Good Luck

Beatle

Awesome find, thank you Paul. I'll give this a shot today. FWIW, mine's headless as well, though yesterday I connected a VGA cable & monitor. It has an APC SMX1500RM2U UPS connected via USB and there are two Apple, dual-port, 1Gbps NICs installed in each PCIe slot. Beyond that it has 10GB of RAM, and 3 stock 2TB ADM's, Hitachi, model # HDS722020ALA330. Not all of the RAM is from Apple/Hynix, etc., but at this point it doesn't matter.

I tried resetting the NVRAM as well, no luck, same results. System Profiler shows the front LED/control panel being connected via USB (I think), so I wonder if this is related:

Frontpanel Controller:

Product ID: 0x8261

Vendor ID: 0x05ac (Apple Inc.)

Version: 0.10

Speed: Up to 12 Mb/sec

Manufacturer: Apple Computer

Location ID: 0x3d100000 / 2

Current Available (mA): 500

Current Required (mA): Unknown (Device has not been configured)

System Profiler shows the front LED/control panel being connected via USB (I think), so I wonder if this is related:

Good catch!

--Paul

Exactly the same thing here, 5 intel xserves, all of them have HWMOND using 87% of one core.

A friend has two Mac Mini servers and they don't suffer from this issue. Im thinking its the LED panel on the front of the Xserve.

More specifically I think it's the interaction between hwmond, the front panel, and, from what paulsuh found, possibly the IOUSB.bundle in his earlier post as of 10.6.8. This might explain the choppy processor LED behavior I thought I saw a few days back. Feels like we're making more progress on this amongst ourselves than Apple is (zing!).

Probably because we have xServes and Apple doesn't ;-)

We have 5 Xserves and 2 MacMini servers here and I can confirm this behavior. The Minis are fine, but the Xserves are using almost an entire processor core.