I had this problem also, and finally solved it.
For me, the issue was that I had files (in my case videos) that the system could not process. How then to find these files?
Well, I searched for all mpg and avi files and converted them to mp4 files - But that still was not working. Then I opened the video-album and minimized the thumbnail-view, looking for grey/black thumbnails indicating that something was off.
That was the key for me, as I then stumbled across some videos with filetype "Apple Intermediate"?? I exported them, converted them to mp4 and re-uploaded them. The location and timestamp had to manually adjusted afterwards, but at least my library is now fully curated.. Finally....
Apple: Please make it simply for users to identify photos/videos that are causing problems... This "treasure-hunt" was long and agonizing for me...
P.S. My photo library is about 50.000 pictures and 2.000 videos