I am getting the exact same thing on my 2017 Highlander Limited with an iPhone 7+ however, my wife's iPhone 6s+ doesn't have this issue at all. If anyone comes across a solution for this please let me know as this is annoying for anyone talking to me.
my iPhone is on the latest version of iOS 11 as of November 2017 and it was also doing the same thing with iOS 10.