Hello Lukasjo,
Just to confirm, are the AirPods disconnecting from your device when this issue happens? Do you receive an error message of some kind? If so, what does the error say? Does this happen with all FaceTime calls, or only FaceTime calls with certain contacts? That information will help us plan how to proceed.
Please check out this link, which has information that may help if your AirPods aren't connecting as expected: If your AirPods won't connect
If you can't connect to your iPhone, iPad, or iPod touch
1. Make sure that your iPhone or iPod touch has the latest version of iOS or that your iPad has the latest version of iPadOS.
2. Put both AirPods in the charging case and make sure that both AirPods are charging.
3. To make sure that Bluetooth is on, go to Settings > Bluetooth.
4. If your AirPods are connected, make sure that they're selected as your audio device. If your AirPods appear in the list of devices but they don't connect, go to the next step.
5. Close the lid, wait 15 seconds, then open the lid. Press and hold the setup button on the back of the charging case for up to 10 seconds. The status light on the front of the charging case should flash white, which means that your AirPods are ready to connect.

6. Hold the charging case, with your AirPods inside and the lid open, next to your iPhone, iPad, or iPod touch.
7. Follow the steps on the screen of your iPhone, iPad, or iPod touch.
8. Test your AirPods. If you still can't connect, reset your AirPods.
If the issue is that FaceTime disconnects, the "Use a fast connection to avoid sound quality issues with FaceTime" part of this link has information and suggestions that may help resolve FaceTime connection issues: If you can't make or receive a FaceTime call on your iPhone or iPad
Use a fast connection to avoid sound quality issues with FaceTime
If your cellular connection or Wi-Fi network runs slowly, or other users stream audio or video on the same Wi-Fi network, you might have these issues:
* Connection alerts or unable to connect
* Choppy video calls
* Black screen
* Dropped calls
Make sure that both you and the person that you call use a fast Wi-Fi or cellular connection. If you use Wi-Fi, FaceTime requires a broadband connection.
Thanks!