There's no different security issues booted into Safe Mode than booted into a normal account. Single-user mode is quite different because then the user is 'root.' However, if all other users than yourself have separate standard accounts, then they cannot make alterations that would require admin or 'root' access unless they know the admin's password. Giving other users admin access or knowledge of the admin's password would be a security problem, and a security mistake.
If you have no other users on the machine but simply wish to prevent a casual walker by from using the computer when it's unattended, then require a password to wake the computer from sleep and/or only operate out of a standard account rather than the admin account.
If you are concerned someone could restart the computer to gain access, then do what Kiraly suggests and use a Firmware Password.
