Siri location reminders problem

So I want to be able to talk to Siri as say: 'remind me when I leave the gym go do this this and this.'

I figured out that to add locations, you need to add it into your contact. I've done that with custom labels, naming them gym, school etc. When I ask where is my gym, it does not know and brings up my contact info with my gym address there. I ask to set reminders for when I get to gym, doesn't know where my gym is, and shows my contact card. Any ideas?

The only labels that work are 'home' and 'work' the rest don't work.