Yeah, I didn't explain this very well: I keep most of my images inside iPhoto, including a bunch of backdrops (924 images, which Ive collected over the years).
They are all part of the same album, "Backdrops", in iPhoto. Now, albums in iPhoto do not correspond directly to a single folder on my drive, so its not a case of simply adding that folder (if only it were that easy!).
You used to be able to pick an iPhoto album on older versions of OS X and then tell it to pick a random image and change it periodically. This feature seems to have been changed and then removed entirely. Try this: in System Prefs -> Desktop & ScreenSaver, select an iPhoto album; you will note that the "Change picture" options at the bottom of the dialog box become grayed out - they are only available for "regular" folders, not iPhoto albums (even though albums are listed there).
Another thing I tried was selecting all my images in that album in iPhoto, then adding my iPhoto Selection folder to the list of folders. As you correctly pointed out, the selection appears to be "cached" and new images will not appear. But in my case, this didn't work at all, I see no thumbnails (maybe it chokes on 924 images? 🙂