can you use transitions with green screen clips?

No. You cannot use a transition with a green screen clip. If you think about it, that would take two renders, and iMovie does all rendering in one pass. So you could do this in Final Cut Pro, which works by rendering as you go, but not in iMovie.

You can simulate this in iMovie by doing your green screen clip as a separate project. Then SHARE it using EXPORT USING QUICKTIME as Apple Intermedate Codec to preserve as much quality as possible. Then import this new clip into iMovie as an event clip. Then you can apply transitions to this new clip. It is cumbersome, but it works.

If you need to do a lot of this get FCP.

Actually, the same problem exists in FCP X. Here is the work around that I discovered along with the person on the help line at Apple:

Create your video as you want it but leaving out the transitions. You should already have applied the keyer effect and remove whatever you do not want, and place into the background whatever the image plate is to replace the keyed out color(s). You should be able to see the foreground and background fine as of now.

Export your video. This compresses everything into one layer as one file (it's like taking a layered image in Photoshop and flattening those layers into one object to create a JPG). Import your consolidated foreground/background video into a new project. Add your transitions.

Now you should be able to preserve the "layers" and have functioning transitions.

You will need to render/export a second time to put the transitions together into one movie file with the visual layers you already created in the first exporting process. Once you do this a second time, all your visual layers and transitions should be flattened together into one functioning movie.