Question: silent mode button slow
Silent mode button slow and less response
That's a mechanical switch and it could be dirty. Gently clean it using a brush, then enable / desable it 3-4 times.
If the issue still persists, have your iPhone inspected at an Apple Store or AASP (they are reopening after the lockdown).
Regards
Giulio