I am trying to copy my thunderbird profile from thunderbird 17 on an older Mac to thunderbird 78 on a new Mac, but cannot seem to get it to work.

You might want to visit the Thunderbird support site with your question. Lots of information there. Thunderbird is an excellent program, in my estimation. In my experience, I think it works like this. You would newly download and install Thunderbird in the new Mac, then set it up with your email provider. This is easy to do. Just follow the instructions to set it up. It has the ability to import addresses, but I assume that is not possible from an external source. If I remember correctly, the old Thunderbird has the ability to export its files and gives you several options as to types of data to use. You can then copy that file to a flash drive, plug it into the new computer and then import the file. I hope this works for you. Good luck

Thunderbird has changed their profiles multiple times since v17 and macOS has probably changed some of the rules & locations of how Thunderbird manages its data & preferences. It may not be possible to import/upgrade directly. @Ronsara's suggestion of exporting the old profile is a good one. If that doesn't work, then maybe you can transfer the Thunderbird v17 data manually and then run the Thunderbird 78 installer to see if it will automatically update the profile for you.

Otherwise, it is probably easiest to just start over with v78.