Cannot use old rescue email address as primary address

I have deleted my rescue email address and then save but when I try to change the primary address with the old rescue email address it says:

This email address is designated as your rescue email address and cannot be used as your Apple ID or Primary email address. Please choose another.

I have tried everything I could.. but it seems that my old rescue email address is still there as a ghost rescue email !

Please help