green phone icon doesn't return to phone screen

When I answer a call, the phone screen disappears. Most of the time, it's a spam call and I want to hang up IMMEDIATELY! But, the little green phone at the top is nonrepsonsive. I can't get back to the phone screen at all and have to wait till the spammer is finished and hangs up. This can't be right?