Can you use a Cat8 10G ethernet cable on the M3 mac

You probably could. However, the Ethernet port on the M3 iMac is a Gigabit Ethernet port. Plugging in a cable that is high-quality enough to carry 10 Gbps traffic is not going to upgrade the M3 iMac's Ethernet port to run any faster than 1 Gbps.

It also assumes that you got the 24" iMac model(s) that have an Ethernet port on the power cable. Base model 24" iMacs do not have this feature.