Howdy,
For random Screen Saver please check if you have any Hot Corners setup for it. I sometimes forget about them until I move the cursor "out of the way" and activate the Screen Saver.
System Preferences > Desktop & Screen Saver > Screen Saver (tab) > Hot Corners
It doesn't sound like it is the Start delay, "Start after" as it is random.
Does this happen with multiple Screen Savers?
Do you have any added software Screen Savers?
You may also want to check the time to Require password once the Screen Saver starts. If set for "immediately" you won't be able to just move the cursor away from the Hot Corner to cancel the Screen Saver.
System Preferences > Security & Privacy > General > Require password [time] after sleep or screen saver begins.
ivan