You can have just one library and have it operate efficiently. The key I've found is to click on an album with only one or two photos in it before closing down. When I next open the library only those few photos need to be rendered making the launch so much quicker. If you are in the All Photos or Photos window when you shut down all of those photos will have to be rendered when opening the library the next time. That can really slow down Photos.
Also keep a minimum of 20-25 GB of free space on your boot drive to facilitate space needed for temporary and swap files. This will go a long way in improving system and application efficiency.
Lastly, with El Capitan I recommend a minimum of 8 GB of memory. Running on just 4 GB is pushing somewhat the envelope for system and applications.
If you have a laptop then you will have some storage issues and it would be difficult to store and run the library from an external HD. It can be done with very small portable EHDs like this one: OWC Mercury On-The-Go Pro
You might have to have your full library on an EHD and have a "traveling" library on your MBP with just the photos you currently need. The paid version of PowerPhotos can be used to copy photos or albums of photos between libraries while keeping the metadata, keywords, titles, faces and locations intact.