Look at the Activity Monitor, System Memory for some hints on whether you should add memory. Here's how to interpret the info:
For video you can put a Apple version fo the ATI Radeon HD 5770 into your machine. It will work in a 2008 mac pro so long as you have Snow Leopard 10.6.4 or greater installed. If you google this you will find plenty of hits of installing a 5770 in a 2008 mac pro.
Note, the articles in the MacPerformanceGuide might be of interest to you.