Action button unresponsive

I have the ultra 4 watch, I have set the action button to workout outdoor walk. When watch has not been used for a few hours, when I press action button there is no response. I have to wake up screen then it starts straight away. I have checked the settings on watch and phone match.

Anyone else have this issue or anyone know how to resolve it.