There's not much you can do. Graphic-heavy games consume considerable system resources. QuickTime screen recording also consumes considerable system resources. When you have both competing for finite computing power, one or both will inevitably lag.
You might be able to improve performance by maxing out the capabilities of your iMac. The first step is generally to add as much RAM as the machine will handle. The next step is to swap the traditional hard drive for an SSD. The step beyond that: a newer, faster Mac than your 2010 model.