I'm having the same problem lately. I have a mid 2010 21.5" iMac with an internal SuperDrive running Yosemite. I tried putting in "Dawn of the Planet of the Apes" and "The Giver" rented from RedBox. Both spin sputter for a bit and then eject the disk back out. So, it never mounts and no software (Finder, Disk Utility, DVD Player) ever gets a chance to see it. Both play fine in a 15 year old DVD player, and my purchased copy of "The Dark Knight" mounts just fine in the iMac. So, I have to assume that something is being done to the DVD's that rental companies are given to prevent them from mounting or something.
Not having any luck on Google trying to confirm that.
I guess I need to amend this. Both DVD's play fine in a early 2011 13" Macbook Pro with the internal SuperDrive. It's running Mavericks still and has a "MATSHITA DVD-R UJ-8A8". The iMac has a "OPTIARC DVD RW AD-5680H". Not sure if it's the drive, the OS, or what (although I will say I'm fairly certain I was having this problem on the iMac before Yosemite, so it's probably not the OS).