Apple Health tracking sleep outside my Sleep schedule?

I've got a Series 7 Apple Watch updated to the latest WatchOS. When I look at my Apple Health data for sleep tracking, on many days it seems to start tracking my sleep at 6pm/7pm, giving me ridiculous sleep totals of 11-12 hours as it somehow counts sitting on the couch or at the dinner table as time asleep.

I have a sleep schedule set in my Sleep focus, with a Wind Down time of 11pm and wake time of 7:30 AM. Why is it counting "sleep" outside of this time? This happens at least 3 days a week, sometimes all 5 days. What is going on?