What would I need to do to check that all is configured correctly?
I would suggest confirming that the AirPort Express Base Station (AX) is configured properly to provide the PPPoE information to your ISP. (ref: AirPort Utility > Manual Setup > Internet > Internet Connection)
Check that Connect Using = PPPoE, and that, the appropriate information is provided for the other fields on this page. On the PPPoE tab page, be sure these fields are correct as well.
I'm not sure if this is the problem though as I've been using the same ISP for some years and my computer (and modem provided by the ISP) has been set up to access this and was doing this automatically without needing further input of credentials.
... and this may not be the issue, but I just wanted to rule out any incorrect settings as being the culprit.
Also, it does connect fine after 10mins
The fact that you are experiencing this after starting up may indicate that the problem may not be with the AX at all, but with the network settings on the Mac.
I suggest, creating a new user account. Log off your current account, and then, log back on with this new account. Does the problem continue?