I have the same problem trying to open them in Quicktime 10.4, but I'm pretty sure it's a bug with High Sierra and not the "unsupported camera" company line. Two reasons:
- when I first set up my new 2017 iMac I used migration assistant after setting up the primary account so I was stuck with two accounts, the second being the restored one with my old stuff under a new username but the first account having the same username as previously. Quicktime opened the AVCHD files in High Sierra under the first account, but gave the "The document BDMV could not be opened" error under the second account. I then decided to do a clean install of High Sierra so that I could migrate my old stuff during installation and have only one account under the original username. That all worked fine but unfortunately now I can't open the AVCHD files at all.
- I can see the video thumbs in the Quicktime clip viewer just fine.
BTW the video files are all on an external drive. When Quicktime opened them OK under the original username the first time, I figured it was some sort of file path issue, but now I've re-installed and there is only one account (with the original username) all the file paths should be the same as before but... no luck.
Unfortunately I don't recall if I tried opening the videos under High Sierra on my old (dead) machine.