Hi,
I've encountered this problem, and though I don't have a fix, I might be able to shed some light on what's actually happening and why OS X can't force-quit the app.
During this summer, I had periodic problems with Entourage when it was unable to connect to my mail server. It would display an error dialog that I couldn't close. If I attempted to force-quit Entourage quickly enough, it teminated, but if I waited too long, I wasn't able to force-quit the app. At this point, things got worse. The longer I waited, the less likely I was able to terminate or force-quit any of my other apps, as well. So I learned that once Entourage got into its error state, it was important to force-quit it as soon as possible. After one of the system restarts I had to do due to this, I took a look in the system console, and saw a lot of messages that reported that Entourage was leaking memory (allocating system memory and not releasing it). This explained why if I waited too long, that Entourage was probably allocating so much memory for itself that there was not enough for any other app (or the operating system itself) to use in order to function. Fortunately for me, the upgrade to 10.5.5 made this problem go away. But if you encounter this problem, it might be useful to check the system console to see what was happening leading up to the time you had to restart the system. It may be that the culprit isn't the app you couldn't force-quit, but some other app that's misbehaving and preventing other apps from getting the resources they need to run properly.
Hope this helps,
Ken