256 MB RAM is just about the minimum that Mac OS 10.4 Tiger needs to boot up and run. So there's no RAM left for the programs you want to run on top of it.
All new Mac minis come with 512MB RAM standard, because Apple realized that they just didn't run well with the original 256MB configuration.
I believe that if you upgrade your Mac mini to 1GB RAM you will see much better operation in all applications, and the elimination of your latency problem with GarageBand and your MIDI controller.
Upgrading the RAM in a Mac mini has to be performed by a qualified technician. It is not something the average user should attempt.
RAM is much cheaper when ordered online compared to purchasing it in a local store.
Do a search at
http://dealram.com.
You need to make sure that you are purchasing genuine Samsung RAM, which is not the least expensive.
Your Mac mini has only one slot for RAM, so if you buy a bigger RAM chip you will have to remove the 256MB chip already in your Mac. You might be able to sell it on eBay for around $10.
You can purchase a 1GB chip for your Mac mini for about $100.
You can purchase a 512MB chip for about $50. But I would go for the 1GB.
Finally, you need to find a qualified technician to install your RAM.