"iTunes could not connect to the iPhone because an invalid response was received from the device" only on my iMac

Could it be a OS problem?

Yes.

iTunes minimum requirements for an iPhone is Mac is v10.9.5 or later and the latest version of iTunes.

It won't work on v10.7.5.

See if your Mac can upgrade to macOS Sierra v10.12.x > How to get macOS Sierra - Apple Support

Then you would able to run the latest iTunes version 12.5.5.5 and be able to sync and backup devices.

An "invalid response" error can mean you are running an older version of iTunes than that required by the iOS version on the device.

iOS 10 requires iTunes 12.5.1 or newer which in turn requires OSX 10.9.5 or newer. Software Update may be misleading because it only checks for updates for the current system version you are running, but that itself may be out of date. It may or may not be possible to upgrade your computer to the required system. Find your computer model on the http://www.everymac.com web site and near the bottom of the system specification section it will say what OS versions it is capable of running. If you cannot run a newer system you will not be able to sync an iOS 10 mobile device to your present computer, though some features may be available with iCloud services such as Apple Music or Match. OSX 10.9 and 10.10 are no longer available. Check this article on El Capitan (OSX 10.11) and Sierra (OSX 10.12) upgrade options https://discussions.apple.com/docs/DOC-10692 These systems may run slower on older machines and require additional RAM purchase (4 GB is a realistic minimum and many recommend 8GB). Doing a big jump in system versions is also more likely to affect old software.

Snow leopard to el capitan, will it cause my macbook to be slow? - https://discussions.apple.com/thread/7412959

Current OSX general upgrade information, including system requirements - http://www.apple.com/osx/how-to-upgrade/