Apple Remote Desktop is off be default. It has to be enabled for some one to be able to remotely connect to the computer. And then, you still have to have a user name and password on the computer to remotely connect with.
If you want to see if remote access has is enabled for Apple Remote Desktop; you can find the setting in, Apple Menu, System Prefrences, Sharing. If it's enabled, Remote Management or Screen Sharing will be checked.
Beucase Apple Remote Desktop Agent is part of the Mac Operating System; even if your not using it, Apple Software Updates will from time to time offer updates for ARD Agent. Software Updates can some times be stacked ontop of each other; so chosing not to install an update, can mean other updates you may want may not be offered. At least until you install the updates those updates require. Also software updates can improve the security of your computer.