Friends disappear from activity sharing!!

So I have an apple watch series 3 and in the fitness app there is the function to share activity with friends. Based on this, my brother got himself an apple watch as well. The issue is: when I send an invitation to my brother it often takes ages until he receives it, however sometimes I have to delete the invite and make him invite me as then I can actually accept the request. So for about 10min I will see his name in my friends list that I share my activity with but whenever I go out and check the progress during the day his name does not show anymore and we cannot see each others activity progress any more. Which means we constantly re-invite each other but the connection never lasts more than 15min max. We restarted, restored and unpaired our devices and looked for any other advise but no one seems to be able to tell us what to do, we also updated our phones and watches but it made no difference.

I got the apple watch simply to hold myself accountable and motivate myself by sharing my progress with my brother. Quite a useless purchase if the main aspect of the activity capture and sharing doesn't work :( Please help!!