This is what: Use safe mode to isolate issues with your Mac - Apple Support says about starting up in Safe Mode...
"To start up in safe mode:
- Start or restart your Mac, then immediately press and hold the Shift key. The Apple logo appears on your display. If you don't see the Apple logo, learn what to do.
- Release the Shift key when you see the login window. If your startup disk is encrypted with FileVault, you might be asked to log in twice: once to unlock the startup disk, and again to log in to the Finder.
To leave safe mode, restart your Mac without pressing any keys during startup."
How to start a Mac in Safe Mode provides details from another perspective, and also describes how to identity the fact that your computer has in fact started up in Safe Mode.
From your description, it sounds to me like you are not pressing the shift key soon enough. My experience with a bluetooth keyboard and Safe Mode results in about a 50% success rate.