Hmm, the System Photo Library isn’t only for iCloud Photos but is the only library that can be used for iCloud Photos.
If you selected to use it, and this was the first time using the Photos App on the iMac, maybe it’s trying to import to that library but the library is syncing to iCloud then running out of storage space. Keep in mind that is just a guess though.
Where are the photos I imported into Photos on my Mac? - Apple Support
This article shows how to turn on, and off, iCloud Photos. Make sure it is not on by accident. Use iCloud Photos to store photos in iCloud - Apple Support
Finally, rather than importing the photos using the Photos App it may be easier to sync the photos on the iMac and iPhone using Finder. Sync photos between your Mac and iPhone or iPad - Apple Support
The difference between importing and syncing is about what happens after a photo is deleted or edited on either device. Imported photos are unrelated files between the iMac and iPhone. So a photo would need updated/deleted on both separately. A synced photo updated/deleted on either device would automatically update on the other device, at the next sync.
Let me know if any of this helps or if you are still seeing the issue.