Yes the method you describe above is going through several levels of conversion.
First the camcorder is converting the digital signal on the tape into an analog signal out of the RCA connectors.
Then your capture card is converting this analog back to digital and then you are converting this to a Mac compatible file.
As you no doubt know the method you used successfully with the FireWire cable plus adapter ,some time previously, was a direct copy of what was on the tape, hence a big difference in quality.
Unfortunately a FireWire to USB adapter will not work for what you want to do here.
Have you tried capturing with QuickTime Player as mentioned by Rich?
Also Rich asked if your Cam' was recognised by iMovie .
Is it recognised but just won't import. Details would help.
What version of iMovie and what OS are you using now.
Can you remember what iMovie version and what OS you used when it worked before.?
Give QT Player a try.This sometimes works where iMovie may not on newer hardware and/or OS.
Rich has covered it above but here is the link from Apple.
https://support.apple.com/en-gb/guide/quicktime-player/qtp356b55534/mac
Read the details in the link but the main points are covered in this screen shot from the link.