The DMG file may be damaged or it may just be so incredibly old and the latest macOS Big Sur no longer supports the older file format. It may contain multiple partitions, resource forks, etc. This was prior to HFS being journaled, etc. This also predates sparse disk images.
However, all is not lost. First of all make a copy of the DMG image file on the newer Big Sur Mac. You might be able to use Unix commands to find out more information and possibly work around the problem. But the built-in Disk Utility may be worthless with very old DMG files.
Open Terminal and attempt this command to write out the debug information of the DMG file.
hdid /Users/<YourHomeFolder>/<PathToFileLocation>/<filename>.dmg -nomount -verbose -debug > disk_debug.txt
The above will scan the file and output all the debugging details about the file to a disk_debug.txt file.
Please reply and attach disk_debug.txt file contents. It may provide insight to the root cause problem.
You might be able to use a similar command to list the unmounted disk and then use the 'dd' command to disk duplicate it to a new physical external disk.
diskutil list
Determine which disk it is, example /dev/disk3, etc. Also determine which disk your external physical drive is.
WARNING! THE dd COMMAND WILL OVERWRITE ANYTHING ON THE "of=/dev/disk5/" BE ABSOLUTELY SURE YOU HAVE THE CORRECT DEVICE OBTAINED FROM THE DISKUTIL LISTING. THIS IS SOMEWHAT DANGEROUS IF YOU SPECIFY THE WRONG DISK YOU WILL LOSE DATA OR IF Y
dd if=/dev/disk3 of=/dev/disk5 bs=131072
Where /dev/disk3 is the DMG image and /dev/disk5 is an external physical drive with nothing you care about on it. The 'bs' is the block size buffer which could be more accurately calculated to give enough buffer room to speed up the process as much as 20 times. However, the value of 131072 should be sufficiently large enough to make a difference in most any modern scenario. If it's taking a ridiculously long amount of time you could abort with Ctrl+C and try again with a 'bs=8M' which gives it considerably more elbow room.
If this works and you manage to be able to access your data files on the external disk, back them up. Then rinse and repeat for any other DMG disk images you may be.
Alternative methods:
- Find an old still working Mac and see if you can access the DMG files there.
- Setup a Virtual Machine (Parallels, VMware Fusion, Oracle VirtualBox). Install an older macOS operating system
- Copy the data from the old Mac or virtual machine to the new Mac.
- Depending the type of files you might run into other compatibility issues with old file formats. For example, finding an application that can actually read antique file formats.