iMessage between iPhone 4S and iPad Air

I have an iPhone 4S for years but haven't used iMessage. But when I recently purchased my new iPad Air I turned on iMessage on both devices to experiment with how it worked between the two. I had my iPad setup using my phone number so that they synced with each other and others would see the message as coming just from my phone no matter which one I used. Well that was fun, but in the end I would rather just use iMessage with my iPad and go back to using standard SMS with my phone. The problem is now anyone else who is using iMessage tries to text me, it would go to my iPad and not my phone, even if I started the conversation with SMS on my phone. I went back to my iPad and switched from using my phone number back to my Apple ID. However, now when an iPhone user sends me a text, it sends from them as an iMessage, but it just dies in space; I don't receive it on either device. Non-iPhone users get through just fine, but it seems like somewhere out there in iOS land, my phone is still seen as an iMessage user even though it's not. Any clues as to how this can be resolved?