The Finder does remember the previews (a.k.a. Thumbnails) for two days of inactivity. So if you go to a folder once it will generate a thumbnail for the files in that folder. If you don't go back to that folder within 2 days, it will purge the cache of that thumbnail. Every time you go back to that folder, after the thumbnail has been created, resets the clock (so to speek) and it waits another two days before purging the thumbnail for that file. If you quit the Finder, all thumbnails are gone.
It sounds like your movies are HUGE!!! Check in the console, see if there is any spew from quicklook saying it timed out. That could explain the slowness, and how some of your thumbnails don't appear, and is a bug in quicklook not the Finder.
One thing you could do to speed things up is open one of your movie files, copy a single frame from it to the clipboard, paste it into something like Photoshop, crop it to 128x128 (or 512x512) and save it as a jpg. Then open that jpg file and copy its contents to the clipboard, then open an Info Window to that file, click on the icon in the upper left corner of the window and press Command-V to paste the image onto the file as a custom icon... There is a bug where the custom icon, after being pasted doesn't always show up in the Finder windows (but it does show up in the Info Window), if you quit/relaunch the Finder it will appear. Custom Icons trump OS generated thumbnails, so if the file has a custom icon the Finder won't try to create a thumbnail for it. Doing this would probably speed up your entire OS because the Finder and quicklook won't be chugging along trying to process your movies every time you open a folder to it.
Hope that helps.