I wonder if you're trying to use
- 32-bit Intel code. That won't run on Catalina or later, even on Intel-based Macs.
- 64-bit Intel code, in the context of an Apple-Silicon-native application. Rosetta 2 can translate Intel code, but it can't handle a mixture of ARM and Intel code. An application must either be entirely native or run completely in translation – no using Intel plug-ins with ARM applications (or vice versa).
- Code that has dependencies on other libraries that are not installed on your Mac.
Looking at
https://www.vamp-plugins.org/download.html?platform=osx
https://vamp-plugins.org/plugin-doc/qm-vamp-plugins.html
https://code.soundsoftware.ac.uk/projects/qm-vamp-plugins/files
I do not see any indication that the Centre for Digital Music has even acknowledged the existence of Apple Silicon Macs, let alone built .dylib files for them.
The description of qm-vamp-plugins-1.8.0-macos.tar.gz as a "64-bit binary for macOS 10.7 or later" does suggest that we can rule out 32-bit code as the cause for the "exec format" error.
It also, at least superficially, would seem to make the "trying to use Intel plugins with an ARM application" scenario more likely. The SoundSoftware site does offer a download link for the QM Vamp Plugins v1.8.0 source code – so if you had software development skills, and Xcode installed on your Mac, you might be able to build an ARM version for yourself.
If you're not a developer, you're probably best off contacting the project and asking them if they plan to release an Apple Silicon version of the plug-ins.