Is there a way to skip Faces on specific albums?

Albums are basically just lists of photos displayed together. Faces can be added to photos, but since a photo can appear in multiple albums, there is no way or anything to be gained by 'skipping' albums.

I want to skip Faces on some albums so as to speed this Photos app a bit...

All photos in your library are automatically scanned for faces, and there is no way to disable this.

You can disable the display of the faces names on the photos, (in the "View" menu > "Hide Faces names"), but that will apply to all photos.

And you can delete the faces thumbnails of strangers from the library. If you assign the name "Stranger" to all faces you don't care about and then delete the "Stranger" item from the Faces tile, these faces will no longer be processed.

I don't know to what extent it applies to Photos, but Apple (oddly) recently updated iPhoto '09 & iPhoto '11: Improving face recognition results - Apple Support