AVPictureInPictureController.isPictureInPicturePossible returns random value

Hello, I am trying to implement PiPs in my custom player. For that, I am starting an AVPlayer and using its AVPlayerLayer to instantiate an AVPictureInPictureController:

// This is in my custom UIView when I setup the AVPictureInPictureController

if AVPictureInPictureController.isPictureInPictureSupported() {
    do {
        try AVAudioSession.sharedInstance().setActive(true)
        try AVAudioSession.sharedInstance().setCategory(AVAudioSession.Category.playback)
    } catch {

    }

    let videoURL = // Content URL //
    let playerItem = AVPlayerItem(url: videoURL!)
    avplayer = AVPlayer(playerItem: playerItem)
    playerLayer = AVPlayerLayer(player: avplayer)
    avplayer!.play()
    
    pipController = AVPictureInPictureController(playerLayer: playerLayer!)
    pipController!.delegate = self

    _ = pipController?.observe(\AVPictureInPictureController.isPictureInPicturePossible,
                            options: [.initial, .new]) { [weak self] _, change in
        self?.pipPossible = change.newValue ?? false
        print("CHANGE pipPossible = \(self?.pipPossible)") // THIS NEVER TRIGGERS
    }

    self.layer.addSublayer(playerLayer!)
}

This is how I try to start the PiP after the user presses a button:

if (pipController!.isPictureInPicturePossible) {
    print("startPIP")
    pipController!.startPictureInPicture()
} else {
    print("PiP is not possible")
}

The behavior I get is that when I run the app, there is a 50x50 change that `pipController.isPictureInPicturePossible` will return true. Therefore, when I press the button, in some runs I see the PiP, and in some other runs I just get the `PiP is not possible` log.

Am I missing something?

Cheers