Prevent people from breaking through my sleep focus by calling twice

My iPhone is set up to block phone calls when I'm asleep. It also has a feature (that I apparently can't control) that lets people break through this by calling a second time. My phone seems to assume this must be an emergency and lets my phone wake me up.

The trouble is that job recruiters and telemarketers have figured this out and are now using this to get through when I don't answer the first time. Since I don't have a standard sleep schedule, these calls can wake me up in the middle of my night.

Is there any way to restrict the people who are allowed to break through my sleep focus? I would be fine if only people in my contact list are allowed to do this, but I can't find a way to configure that. Is this possible?