How to not have other Apple users send me iMessage instead of SMS

I have an Android phone I use for texting.

I have an iPad I use Wi-Fi only at home. I have iMessage turned on so I can use those features at home like sending photos, voice memos, etc with just a few people.

My issue is, other users with an iPhone will text me and the texts are only sent my iPad via iMessage. I don't want this because I won't get those messages for hours since the iPad is at home. I want all my texts to go to my Android phone. I assume their contact listing for me includes the email my iMessage uses.

Is there a setting I can change in my account to prevent others from sending me iMessage instead of SMS, without turning iMessage off altogether?