Enabling IPv6 breaks Apple Music on HomePod mini

I have a problem that occurs when I want to play certain radio stations through Siri on my homepod mini. If I ask Siri to play e.g. "NPO Radio 2", Siri will respond with "There was a problem with Apple Music" after around 10 seconds and no music will be played. Other radio stations do play normally. I noticed that when I use the homepod mini on another network at another location, the issue would not happen.

I have a network with several VLANs. On the regular LAN, I have two homepod minis and an Apple TV 4k. I run pfSense on a Netgate 6100 with the latest (at the time of writing this post) firmware 23.09.1.

After trying to figure out what it is, I finally figured out it is linked to IPv6 being enabled on my LAN. When I turn off IPv6 functionality, everything works as expected. As soon as I enable IPv6 again, the same issue appears. Base IPv6 configuration is that I get a PD of /48 from my fiber provider KPN. I track the interface on my LAN. I have setup Router Advertisements in Assisted mode as per one of the many tutorials for my provider on the internet.

I read somewhere that there could be a priority clash between Router Advertisement messages coming from the Apple Matter/Thread networking functionality and that from pfSense. I tried heightening the priority for RA messages in pfSense from normal to high. That does not resolve the error.

I am looking for suggestions on what could be the root cause for this issue. I want to figure out a way of having both IPv6 working on my pfSense and have the Apple devices behave as they should. I know I do not need IPv6, but hey, I'm a hobbyist in search of trouble ;-)