Three Way Calls Merge/Swap

I have an iPhone 7 and am using it for phone calls much more working from home with multiple calls at once. If I have someone on the phone and I add a call and the second caller doesn't pick up, I can't end the second call without hanging up on the first caller. Also, my Swap button is grayed out in this scenario. Any thoughts?