Hello CaptSaltyJack,
Welcome to Apple Support Communities. I understand your Mac mini hasn't been going to sleep lately and you think it's because of the Handoff feature. I am happy to help.
With Handoff, you can start something on one device (Mac, iOS, or Apple Watch) and then pick it up on another without losing focus on what you’re doing. For example, look at a webpage on your iPhone, then pick up where you left off in Safari on your Mac. You can use Handoff with many Apple apps—for example, Calendar, Contacts, Pages, or Safari. Some third-party apps may also work with Handoff."
I don't think that it is the handoff feature that is preventing it from going to sleep. Try the steps in the article below to put the Mac to sleep.
If your Mac doesn't sleep or wake when expected - Apple Support
This article might be helpful as well for Sleep mode: Mac computer Sleep Indicator Light behavior
Let me know if you have any questions.
Have a great day!