Voice Control / Dictation consumes 600% CPU even when not listening

MacBook Air M2 2022, macOS Sequoia 15.0.1

This is what happens after using Voice Control for some time. This is when voice control is in "Stop Listening" mode, so it should be pretty much idle:

When I switch off Voice Control in System Settings, it starts nicely:

However, at some time later it starts eating up CPU. I do not yet know what triggers it. The same behaviour exists on an M1 MacBook Air.

Any ideas what could cause this?