How do I transfer my Photos to a new machine?

Your new Mac will ask you if you want to transfer anything from another Mac, PC or a backup during the first startup, so just follow the steps. See -> https://support.apple.com/en-us/HT204350

If you have to transfer a huge amount of data, I would recommend you to connect both Macs with a FireWire or Thunderbolt cable (depending on your MacBook Pro model, and you may need adapters) or make a backup of your old Mac with Time Machine and use it on your new iMac to transfer your photos. This is because Wi-Fi and Ethernet transfers are very slow.

Put the iPhoto library onto an external HD or if it fits onto a flash drive you can use that, then connect the media to the new machine and drag it over to ~/Pictures, the ~ is your home folder.

If you want to migrate all the apps, settings and files on the MBP then back it us using Time Machine and after it's done backing up, connect to the iMac and run Migration Assistant (Applications - Utilities - Migration Assistant).

Likely because you did not follow the directions provided. What file format are the files in on the MBP? If they are JPG or Raw then then will come over as such assuming you do a simple drag and drop. You can also connect the two machines together using Firewire Target Mode however you will need a Thunderbolt cable and a Thunderbolt to FW adapter and that assumes the MBP has FW.

Setting-up a new Mac from an old one, its Backups, or a PC

If your old computer is not recognized by Migration Assistant you can connect the two computers and transfer files. This can be done with a Firewire connection running the old computer in Target Disk Mode or via Wi-Fi with both computers running normally.