AirPods Pro 2 Conversation Awareness

I just got my AirPods Pro 2 earlier today and used it for a bit for testing.

I listened to music for a few hours and tried Conversation Awareness and found that if I hold a conversation for more than a minute, the volume sometimes spikes up and down before it ‘adjusts’ properly from low to high volume. Is this normal?

Do note that I am in a fairly noisy environment with a lot of people talking, but it doesn’t pick up the noise from the environment enough for the CA to activate