There may be two reasons -
- either your Mac is running out of memory, because you have too many application windows open, or you have a program running in the background that is hogging all the available memory,
- or your GarageBand project is too ambitious - too many software instrument tracks with complicated effects.
With reason 1 it will help to save your GarageBand project, quit all running applications and to reboot. Then launch GB again, but refrain from running other applications in the background, i.e. Safari and multimedia apps like iMovie, iPhoto, Powerpoint will need a lot of memory.
For ways to simplify your project without losing quality see: http://support.apple.com/kb/PH1874
If you're using a MacBook, be sure it's not running on battery because the power conservation features try to minimize disk activity. I get this error every time I forget and try working unplugged. I've not see this posted elswhere on Apple discussions. I found the tip on a blog somewhere and it fixed my problem, so I thought I'd mention it on an Apple post.
leonieDF is of course right, but you can still get this error on a small 2 or 3 track project even if you have all other apps closed, especially if you are powering a USB audio interface while running off your battery.