Hello, hboyer9!
Thanks for reaching out to the Apple Support Communities! It sounds like you're having issues with the restriction to contact specific contacts during downtime not working properly. Is there a message on the device to "Ignore Limit" when an unauthorized contact is contacted?
We'd recommend taking a look at this article to Use Screen Time on your iPhone, iPad, or iPod touch just to confirm that the settings are set to your preferences. You'll want to make sure that you have enabled the "Block at Downtime" / "Block at End of Limit" options for the phone and that your Screen Time passcode isn't known by the person using the device. In addition to this, double check to make sure that the app hasn't been included in the Always Allowed section of the Screen Time settings.
Let us know the results and have a great day!