Firewire 800 Plug is Too Big for Socket in Back of my iMac

The square-ish 9-pin Firewire 800 should be correct for a Late 2009 - Mid 2011 iMac. (Make sure you aren't trying to connect to the Mini-displayport)

Late 2012 and newer iMacs do not have Firewire ports and will need a Thunderbolt to Firewire adaptor cable.

The cable you bought, per picture: is Firewire 400 <--> Firewire 800. This cable is usually used in older hardwares.

It might be helpful to know your iMac Model.

For 2012 - 2015: Thunderbolt 2 capable - that means you have to get this adapter (also see picture below).

For 2017 to present: Thunderbolt 3 - that you have to add one more adapter (in order to achieve: thunderbolt 3 -> thunderbolt 2 -> Firewire 800)

There are few third party hardware makers for making thunderbolt 3 --> hub (that might include firewire port). I have never tried those, therefore, I can not comment on those.

Edit: I did have used Thunderbolt 2 -> Firewire 800 adapter, then added "Firewire 800 - Friewire 400" cable for my external optical drive for 2012 iMac and 2014 Mac Mini. Right after, I have merged with USB 3 external devices (TimeMachine and optical drive) as they (USB 3.0 / 3.1) seems to be more future proof (I hope).

You are very welcome..