Apple Milanese band is no longer holding tight

I have the black SS Milanese watch band and the magnet is not holding tight to the band anymore. It's never been submerged in a pool or shower, only sweat and cleaned with Dawn, water, and a toothbrush and let air dry maybe 4 times a year.

I was thinking maybe it's the rubber seal around the magnet, it looks fine but if it is hardening it would slide instead of gripping with the magnet.

Any ideas on how to get this fixed?

Thanks