I'm not sure if my computer is not powerful enough for such a task, or if quicktime 10.0 isn't capable of smoothly recording gameplay.
I've tried lowering graphic settings in the game and switching the recorder to medium quality, but no matter what the video is still choppy, distorted, and skips.
The main problem is that the CPU power is divided between both the rendering of what is displayed on the screen and simultaneously recording that screen redering. Further, the app is programmed with a more or less fixed recoding data rate dependent on the recording quality setting and screen area to be recorded. Basically, the "scratch" file captue data rate on my systems may vary from a low of about 20 Mbps to a high on the order of nearly 80 Mbps. Since the recoding data rate is more or less fixed for each recording session, the QT X screen capture routine varies the frame rate in response to the amount of CPU power that is available for recording. Thus, if you check the playback frame rates for your files, you will probably see relatively low fps which produce the "chopply" playback. You will probably also note that the lower quality recording setting produces a somewhat higher frame rate with slightly smoother playback than the HQ mode. That is because the Photo JPEG medium efficiency codec is less CPU intensive than the MPEG-4 AVC (H.264) high efficiency video codec which produces better video quality in a smaller file package.
The normal method for improving the smoothness of final playback is to decrease the area of your scratch recording which, in turn, tends to increase the frame rate of your scratch file providing a smoother playback potential. Some apps allow you to select and record a given area of the screen at its current resolution and/or record the display at a "scaled resolution" to reduce the surface area actually being recoded. Other apps only record the screen at its "native" resolution with or without the ability select a specific area of the screen for recording. Still others may allow you to record "objects" such as a gaming window which may or may not already be scaled by the user to selected fraction of the entire display area. In short, what you can do to increase the frame rate and smooth playback may vary drastically depending on the specific screen capture app use. QT X v10.2 allows recording of both a user selected area or a "scaled" display rendering to improve the frame rate but I rarely use it. I would normally use the moderately priced ScreenFlick app instead for simple projects or an app like Snaps Pro X or ScreenFlow for more complex screen capture projects.