If the computer was not ordered with a different drive option the the entry level, or has not been treated to an aftermarket faster drive solution, it is your hard drive. Apple uses a slow, 5400 RPM drive rated at SATA 3GB/sec in a computer with a fast SATA 6GB/sec drive bus. Adding RAM will make zero difference in your boot time if you still have the slow factory hard drive.
I have the same computer and upgraded it to a solid-state drive at home about three years ago. To demonstrate how RAM and then a fast SSD affect app load times. I have this data from my MBP13 2012:
Base system as shipped:
4GB RAM and slow SATA 3GBps 5400rpm hard drive: Office 2008 and Photoshop Elements 12 took 15-18 seconds to be ready to use.
First upgrade, RAM:
8GB RAM and slow SATA 3GBps 5400rpm hard drive: Office and Photoshop Elements took 15-18 seconds to be ready to use.
Second upgrade, inexpensive solid-state drive
8GB RAM and fast SATA 6GBps SSD: Office and Photoshop Elements take under 4 seconds to be ready to use.
The boot time is now quite fast but I've not checked in lately