Particularly the description of Inactive memory.
This information is in RAM but it is not actively being used, it was recently used.
For example, if you've been using Mail and then quit it, the RAM that Mail was using is marked as Inactive memory. Inactive memory is available for use by another application, just like Free memory. However, if you open Mail before its Inactive memory is used by a different application, Mail will open quicker because its Inactive memory is converted to Active memory, instead of loading it from the slower drive.
I agree: unless your purge cache memory using the "purge" command in Terminal, or a third-party conveniently named Purge, whatever memory a recently closed app has used is still preempted by it, and remains available. The same goes in Windows, when DLLs remain in active memory throughout a whole session, and sometimes clog the system to a halt. Memory management in OS X is much better, though...
The only way not to experiment these slowdowns is to put your Mac to sleep instead of shutting it down. You can decide to hibernate it, too.