I can uninstall by going to Library/Audio/Plug-Ins/Components to remove the file. However, my Components directory is completely empty,
the path "Library/Audio/Plug-Ins/Components" is the root directory, is that where you really looked, or was it "~/Library/Audio/Plug-Ins/Components", from your "Home" directory? which ever you looked at, check the other (it's most likely in the root directory)
Have you found the component by now?
I made a trial install of voxReducer2 to checkk where it gets unstalled - the only install option was in the user library, not the system library for all users. The plug-in ended up in ~/Library/Audio/Plug-Ins/Components/voxReducer2.component/ - in the hidden user library. That is very unusual. As HangTime said, usually these plug-ins will be installed in the System Library for all users.
To remove it from the user library, you need to open the hidden library. Use the Finder's "Go" menu:
Click somewhere on the empty Desktop to bring the Finder forard, then Select "Go" from the main menu bar and hold down the alt/options key ⌥, until "Library" appears in the drop-down menu.
Then select Library, and in the window that will open, go to the "Components" folder.
You need really to get rid of his plug-in in Mavericks - I only used it for a few minutes and Garageband needed to be force quit!