After a week of testing with no graphics glitches or crashes whatsoever, I think it's safe to say I've finally fixed the problem on my machine. The "fix" is basically a do-it-yourself firmware upgrade for the graphics card, in my case the ATI Radeon HD 5750 (really a Mobility HD 5850) inside my mid-2010 iMac. It's a fairly straightforward fix, although it does require you to use a Bootcamp partition to run Windows on your machine and it's not for the faint of heart. Flashing the BIOS (firmware) of a card (or anything else) can be risky. If something goes wrong during the process you might not be able to boot your machine. Chances are everything will be fine, and for me it was a very smooth process, but you should always be prepared for the worst.
Be aware, too, that this is a fix for a very specific problem. Ever since the Lion upgrade, my iMac has been experiencing graphical glitches, especially in Finder windows (Quicklook previews were a disaster), although they also started spreading to Mail windows and Safari menus. I kept hoping a Lion update would fix it; instead, it kept getting worse. With 10.7.4, I also began to experience the crashes that others have described in this thread, where the screen would freeze (usually while showing odd square-shaped artefacts) and only the mouse pointer would move. A hard re-boot was the only fix. Console logs showed this was a problem with the GPU (GPU Debug Info messages appeared every time).
I'd always assumed the problem was Lion specific, but a post in this thread showed me the crashes could occur in Snow Leopard, too. So I started digging around and it turns out the exact problem also appeared for users running the same ATI card in Windows machines. Lion may have revealed the problem, just as Windows 7 apparently revealed the problem for PC users, but it's really a problem with the GPU. In a nutshell, the problem is this: the BIOS of certain ATI HD 4XXX and 5XXX cards sets the power-saving idle clock speeds too low. Whenever the GPU is required to switch to a higher speed (for example, by drawing a Quicklook image preview), glitches occur and stability is compromised. I think Lion (and Windows 7) allows the GPU to enter the idle state way more often than Snow Leopard did, hence the increased frequency of the problem.
Check these links for a discussion of the problem:
http://www.tomshardware.com/forum/276896-33-radeon-5750-display-driver-stopped-r esponding
http://www.tomshardware.com/forum/283018-33-5770-crashs-freezes-colors-card
http://www.techsupportforum.com/forums/f24/ati-sapphire-5770-problem-442797.html
http://www.tomshardware.com/news/ATI-Radeon-Gray-Screen-Crash,9529.html
On the PC side, ATI driver updates and in some cases BIOS (firmware) updates have fixed the problem. You can also use ATI's Catalyst Control Centre on a PC to tweak a card's core and memory clock speeds without having to flash the BIOS.
I'm not sure that a driver update from Apple will fix the problem. There have been plenty of updates to the ATI kext files since Lion's release and the problem remains. It may be that the drivers can't override the speeds set in the BIOS, in which case a firmware update for affected cards is needed. But Apple doesn't appear to be in any hurry to release a graphics firmware update for mid-2010 iMacs, which is why I decided to do it myself. I was having so many problems that my machine was basically unusable, and I figured I'd need a new graphics card anyway if the fix didn't work, so what did I really have to lose?
As luck would have it, the tools to extract, edit and re-flash an ATI graphics card's BIOS are readily available to Windows users. There's also an excellent guide to the entire process, which I read and followed step by step. The guide is here:
http://www.techpowerup.com/articles/overclocking/vidcard/154
Essentially, I booted into Windows 7 (that's why you need a Bootcamp partition - virtualisation software won't give you direct access to the hardware) and followed the Techpowerup.com guide.
Here's what I did:
First, I extracted the BIOS using GPU-Z (available here: http://www.techpowerup.com/downloads/2137/TechPowerUp_GPU-Z_v0.6.2.html).
Then, I loaded the BIOS file into Radeon BIOS Editor (RBE), available here: http://www.techpowerup.com/rbe/
Radeon BIOS Editor gives you all sorts of interesting information about your GPU's BIOS, but the most important tab for our purposes is Clock Settings. The Clock Settings tab shows the clock speeds and voltages of the various PowerPlay settings (the boot mode and various power-saving modes, for example) on the card. The GPU speed (in MHz) is the core clock speed; the RAM speed is the memory clock speed.
Sure enough, the lowest power-saving setting of my card was set to 157 (GPU) and 295 (RAM) - the same setting that had caused problems for Windows users. So I changed it to 398 (GPU) and 1000 (RAM), which matched the next lowest PowerPlay setting in the card and was also close to the suggested fix from PC forums (from 157/300 to 400/900).
I then saved the modified BIOS and flashed it the card using ATI WinFlash (available here: http://www.techpowerup.com/downloads/2107/ATI_Winflash_2.0.1.18.html), rebooted back into Windows to confirm the changes had taken, then rebooted back into OS X.
I've been running glitch-free ever since - no graphics artefacts, no crashes, no problems of any kind. It's only been a week but I'm certain it's a permanent fix. Before I flashed the GPU I'd had three consecutive overnight lockup and I'd been having problems with glitches and graphics artefacts every day. Every time I clicked on a Finder window I'd regret it; every time I opened an email with an embedded image I'd flinch. If the problems I'd been experiencing for months on end were going to resurface, I'm sure they would have resurfaced by now.
I've been running Hardware Monitor to keep an eye on the temperature of the graphics card heatsink and diode. Obviously the idle speed of the GPU is slightly higher now, but the temperatures are well within the normal range so I don't expect any problems to arise. Right now, everything "just works".
As I said, this is not a fix for the faint of heart. If you're not comfortable with tinkering with your hardware, don't do it. I've made Apple engineers aware of it and hope they'll release their own firmware update for affected cards.