Apple watch insists on using iMessage

My App watch series 5 insists on using imessage even for Android contacts. I.e. The same contact use Sms text on the iphone, but the watch keeps trying to send as imessage and hence will fail.

The watch has cellular. On the iphone I only have imessage enabled on the apple id, not the cellular number. I have tried restarting both phone and watch. If I disable imessage completely on the phone, messages will work just fine on the watch.

the setting to use sms when imessage is unavilable is activated as well.