How can I answer my phone when not wearing my hearing aids

Without more information about your hearing problems it is difficult to offer real solutions. It will depend how severe is your hearing loss, but the obvious workarounds might be to try answering and shifting to speakerphone with more volume, or if that is not enough then play for time by responding with a text message saying you will call back when you have your hearing aids back in action.