face id unlocking when im not paying attention

does anyone elses face id unlock and open your phone when you dont want it to? i dont really know how to describe it well but basically whenever my phones in my hand and im just walking or doing whatever ill look down and all of a sudden my phones unlocked and usually on an app or texting someone. is there a way to make face id harder? because id rather not remove it all together but its starting to become an issue