All Memoji Apple Watch Face not working

I have created two Memoji on my iPhone and both of them are available on Stickers on my Apple Watch.

I can select either one or the other as my Apple Watch Face, however when I select "All Memoji", my Apple Watch just switches to the Mouse Animoji.

This happens both when I setup the face of the Apple Watch directly on the watch or via my iPhone.

This has always happened after an update, but "after sometime", the Memojis "came back". Last week, I had to fully reset my Apple Watch, restored the back up (the Memojis are there), but can yet setup "All Memoji" Apple Watch Face properly.