Sadly even when starting fresh, things seem to lock up, current state:
iOS: 127 photos (57 downloading)
MacOS: 127 photos (1 downloading)
iCloud: 127 photos
So it seems that iCloud holds the right pictures but the devices don't get fully in sync when "downloading originals" is enabled.
Also, even with 4 small albums, when I deleted one image the MacOS Photos crashed.
Happens since the synchronisation is stuck.
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000611ed91c2ff0
Exception Note: EXC_CORPSE_NOTIFY
Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [2578]
Will keep on trying different scenarios till this is fixed or has a clear solution.