The Guest Network feature depends on the AirPort being configured as the main router for the network. To do that, the AirPort is expecting an Internet connection from a simple modem......which will have only one Ethernet port.
The new "modem" that your provider has furnished you is actually a modem/router, which is also called a gateway by some folks. This type of device is actually two products in one, since it combines the functions of a modem and a Wi-Fi router in one single package.
So, you actually have two routers on your network, when you only need one.
Since the AirPort Extreme and AirPort Utility "sees" another router on the network.....(the new device that you call your modem).....and you do not want to have two devices......the modem/router and the AirPort both trying to act as routers for the same network, the correct setting for the AirPort is Bridge Mode, which Apple calls "Off (Bridge Mode)".
I'm not recommending that you do this, but you could try to leave the AirPort set up as router to see how things behave. This will create what is known as a Double NAT error on the network......something that you would normally want to avoid, since it can result in slow connections, slow browsing (or, no browsing on some sites) and other problems.
If you find that the Double NAT error is not causing issues on your network, there is a setting in AirPort Utility to tell the AirPort to "ignore" the error, so that it will display a green light. This won't make the error go away, of course.
If you want to configure things correctly, set up the AirPort in Bridge Mode and then use the WiFi on the new "modem" as your guest network.
The simplest solution of all would be to see if your provider can swap out the modem/router for a simple modem. In some case, it is also possible to configure a modem/router to operate only as a simple modem. Your provider may or may not be able to help on this.