GPU rattles during TensorFlow training

Question

GPU rattles during TensorFlow training

Hello,

I've noticed recently that when I am training a (rather large, 12 million parameters) model on TensorFlow that if I listen closely, I can hear the GPU making a quiet clicking/rattling sound. Should I be concerned at all? The temperature stays around 60 Celsius so I don't think that is an issue, and the fans are operating normally. I have a new MacBook Pro with the M1 Max chip. Any advice would be greatly appreciated.

Thanks everyone

MacBook Pro 16″, macOS 12.0

Posted on Jan 18, 2022 1:28 PM

Reply

Answer 1

Jan 18, 2022 4:47 PM in response to Mediocrates1208

Rattling suggests that moving parts are moving about, and something might go flying off. But that is not what is happening here.

Once you get away from thinking of it as rattling, and think about it as computing so fast it becomes audible, it starts to be pretty cool instead of alarming.

Reply

Answer 2

Jan 18, 2022 3:09 PM in response to Mediocrates1208

Would you consider an alternate diagnosis?

Listen to this sound:

SSDs are NOT SILENT! SSD M.2 making high pitch noise or crackling/whirring

https://www.youtube.com/watch?v=KS-BHI667po

.

Reply

Answer 3

Jan 18, 2022 2:51 PM in response to Grant Bennet-Alder

As soon as the training bar appears and my Activity Monitor shows maximum GPU usage from that Python process, I can hear the quiet rattling. However, the rattling does seem to disappear after a few minutes while training carries on as normal. I don't think there is much of an issue, but I just want to double-check.

Reply

Answer 4

Jan 18, 2022 4:25 PM in response to Mediocrates1208

The SSD noise is not down to moving parts -- there are none.

Maybe your original assessment is correct -- that the data crunching in the GPU is happening in just the right way that certain patterns of data crunching are in the audible range.

Reply

Answer 5

Jan 18, 2022 2:03 PM in response to Mediocrates1208

That is a remarkable diagnosis.

How did you decide that?

Reply

Answer 6

Jan 18, 2022 3:56 PM in response to Grant Bennet-Alder

Hmm, although there is a lot of data I am not sure it is the SSD as the rattling is consistent and not scratchy like the SSD. Also, no such sound occurs when I shuffle the data with the CPU.

Reply

Answer 7

Jan 18, 2022 4:43 PM in response to Grant Bennet-Alder

Ok. Do you think that is something I should be concerned about?

Reply

Answer 8

Jan 18, 2022 5:01 PM in response to Grant Bennet-Alder

That's a great way to think about it. Thanks for your help!

Reply

GPU rattles during TensorFlow training

Similar questions