It may be your Login Items interfering with existing Sleep Mode. Login Items are one of the processes Safe Mode takes out. You can try going to System Preferences > Users & Groups > Login Items and remove them all using the ➖ button. (You always add them back with the ➕ button.)
If it's not that, then I don't know. Sleep problems are some of the most difficult to diagnose.
FWIW, I gave up on the Sleep function years ago. Ever since the inception of OS X there have been Sleep related issues. And even more lately. I just use Display Sleep and keep it set for 15 minutes so I'm not burning the screen. That way your Mac uses very little power over Sleep and is ready to use at the touch of a key without any grogginess. UNIX based systems were made to be always on and they’ve been running busy servers 24/7 for decades.
EDIT: We happen to have identical iMacs and I've been running mine w/o Sleep since it was new. I rarely shutdown or restart except for updates and upgrades. It's as good as when it was new.