Any downside to merging two photo libraries by importing

I have organized most of my photos into albums on my System Photo Library. However I have a 2nd photo library that is quite large. A large number of the photos in this 2nd photo library are already in the System Photo Library - but may have different meta data so will be considered "new" and will still be imported. Is there any reason I wouldn't want to import all of the "new" photos in the 2nd photo library into the System Photo Library? I realize I will then have a very large System Photo Library. But I gather that all of the photos that are imported will just wind up in the "Library" of the System Photo Library - none will wind up in any Albums (unless I specifically put them in an Album) - so I won't have any of the duplicate photos showing up in Albums. So - is there any harm in essentially merging the two libraries that I am not thinking of? Thanks.