Can't edit Remote Management users

I have a OS X Yosemite server setup on a mac mini, with remote management enabled and a couple of users defined under "Allow access for" in the system preferences. I need to remove some users, as they should not be allowed to connect remotely using SSH or Screen Sharing, but the "minus" button is disabled and all items under "options" are grayed out (note that I'm logged in with an administrator account).

I thought that the server app had something to do with that and disabled the Screen Sharing and SSH for those specific users under Access to services in the server preferences. This worked for SSH, but they are still listed in the system preferences and can still access with vnc://servername .

I'd really appreciate some help on this.

Thanks