Turn off swipe up on banner to decline incoming call

I tend to accidentally decline incoming calls when I’m using my phone and a call banner at the top pops up. When I try to press the green phone button to accept I always accidentally swipe up and it declines the call. How do I turn this feature off?