The first thing to try is to backup your Mac and to update to the most recent version of El Capitan - MacOS X10.11.2, if you have not already done so. That fixed a few bugs in the Photos.app.
Photos needs to do a lot of background processing when you first open a library, and it is trying to get all the memory it can get. Have you tried to let Photos run over night with the problematic second library open, so the photos model can finish its tasks, while you do not need the mac for other things?
If letting Photo run over night unmolested does not get the mac back to normal, check the Console app, if you are seeing repeated error messages, related to Photos. If you are seeing something suspicious post it here.
Sometimes one single photo or video in your library, that cannot be processed an be holding up the background processing.
Check your problematic library for photos or videos in an unusual format. Create a smart album with the rule "Photo is unable to upload to iCloud Photo Library". This album will show all items in your library that are having an unsupported format. Remove these items from your library by exporting the original files to a folder on your Desktop and then deleting these items from the library.
If all items are supported, but Photos is still hanging, try to repair the library: See this Help page: https://help.apple.com/photos/mac/1.0/?lang=en#/pht6be18f93