Apple Pay shipping address issue

I have an issue with the shipping address attached to my Apple ID whenever I try to use Apple Pay. I live in a flat and so my address has both a street number and a flat/apartment number. When I try to use Apple Pay the flat number gets removed from the address details on the payment sheet and only the street number is left. I first noticed it when I bought my AirPods and saw the delivery driver drive up to my building then drive away again without even getting out of the van and when I went to look at my invoice I noticed the flat number wasn't included so he managed to get to the correct building because he had the street address but didn't know which flat to deliver to, so he left and I had to contact apple to get them to update the flat number with the delivery company. I went to buy something today using Apple Pay and the same thing happened but this time I noticed the flat number had been removed before I completed the payment.

I've gone into my payment/shipment settings in my Apple ID and the flat and street number are both there, I've tried putting them both on separate lines in the address fields but still when I try to make an Apple Pay payment only the street number shows up on the payment sheet before I use the Touch ID to confirm the payment so I have to cancel it or I would end up with the same problem I had with the AirPods delivery.

Sorry a bit long winded but anyone else had this issue?