Sleep mode turns off in the middle of the night, so I get notifications after about 4:30 AM

I sleep with my new Apple watch SE because I use the alarm (so much nicer than the buzzing of a real alarm. If I manually set the watch to sleep mode, it will go out of sleep mode at some point in the middle of the night and I will start getting notifications. I did some research, and it appeared to be a "feature/bug" in which it would not stay in sleep mode for longer than 6-7 hrs. So I set up a sleep schedule on my phone as this was the recommended fix. It does go into sleep mode when my phone does at 9PM, but it consistently starts notifying me at 4:30 AM every morning. My phone is still in sleep mode. When I go to sleep, it shows the little sleep icon, but when it wakes me up at 4:30 there is no sleep or moon icon. I'm about to return it and go back to my fitbit which did not have these issues (although it had plenty of other issues). Help!