This is empirical, but in my case, it appears to be rebuilding an index of photo file names. I have a fairly large number of photos (>10000) that I share to apple TV. Any time I change anything in a photo folder (add, remove, move), I have to then go into the sharing section of system properties, unchoose the affected folder(s), apply, then rechoose the same folders and apply. This is necessary to get the changes to show up on the Apple TV. As soon as I go through this routine, the AMPLibraryAgent starts up and monopolizes the CPU for at least an hour. During this time, I cannot access any photos on the Apple TV. As soon as the library agent finishes, I can see my changes.
I believe this is necessary so that the apple tv can randomize the viewing order of photos. Apple doesn't send the library to the TV unit, just the index file, which the TV box then uses to point back to the library in whatever order you choose.
It shouldn't take this long to rebuild a database index, so maybe it's something else. Like I said, this is empirical. But it consistently behaves this way. The same thing used to happen when the photo sharing was managed from within iTunes.