Home sharing, 2 macs, and Remote on iPad

I have two macs both running Home Sharing with the same apple ID. I have Remote set up on my iPad.

I have my main music library downstairs on my mac pro. This is played on my mac mini upstairs via Home Sharing.

I want the iPad Remote to control the music by playing it on the mac mini.

The first time I used the iPad Remote, it worked fine, but later in the day it played the music on my mac pro instead. How can I tell it to play through a specific computer?