How to remove an old phone number associated with Apple ID

I recently changed my phone number (same mobile carrier). For some reason the old number is still associated with my Apple ID. When I go to Settings, select my name at the top, select "Name, Phone Numbers, Email", my old number is listed with my new number and email. When I select "Edit", it doesn't allow my to remove the old number. It only shows (i) next to it. It tells me to sign out of iMessage and Facetime. I signed out of both, then signed back in, but the old number wasn't removed. I then changed my Apple ID password, but Apple ID still shows my old number along with my new number. I don 't know what else to do. How can I remove the old phone number from Apple ID?