How to disable automatic speakerphone on facetime calls

We usually call each other with facetime audio. I usually answer just in my ear if I don't have any headphones connected. Sometimes I'm in an environment where it is not really good etiquette to have your phone on speakerphone (open plan office). When people call me, sometimes they either want to show me something or they just realise they wanted to call with video and switch to video facetime and that does not ask me if I want to do that and turns the speakerphone on and that starts blasting the whole office. I started telling people not to do that because it is annoying - but I shouldn't be relying on people not doing that to me in the first place. Is there a way to make facetime ask me if I want to turn it on speakerphone? I feel like it should be my choice whether I have it on earpiece or loud speakerphone and the other person should not control that.