Directory Utility is on your Mac already and can do this:
From the Finder use the "Go" menu and click "Go to Folder..." or use Command + Shift + G
Paste this in:
/System/Library/CoreServices/Directory Utility.app
FIrst Find the Users you still use:
On the left sidebar you can find your users and their GeneratedUID is the field you are looking for on the right when you click on each user.
Copy/Paste the UIDs of each user you use to a Stickie or somewhere you can paste text for a moment.
We want to note them so we don't delete the wrong one, although I am not sure that would really hurt anything. You could probably just go back in manually and add what you want.
Finding the groups:
Up near the top toolbar you will see a couple drop down menus completing a sentence that says "Viewing <Users> in node </Local/Default>",
Change "Users" to groups, click the lock and authenticate as an admin user.
Choose either "Remote Login ACL" or "Screen Sharing ACL" from the left and delete out the entries you didn't save before in the copy/paste. Don't forget to change both the "GroupMembers" and the "GroupMemebership".
Click "Save" and you are done!
(If anyone really wants the command line way to work, try com.apple.access_screensharing instead of com.apple.access_ssh, although I really recommend some sort of GUI for this unless you are familiar with such things and were just looking for the right file.
Edit: Oh and check "SharePoints" instead of "Groups" if you are having this issue on a shared folder. The folder will list the groupID but also the groupName, which may be the folder name, on mine it was not but likely because of a rename or copy. Then swicth back to Groups and find the group, repeat the steps from clicking the lock and on.