I actually saved to a network folder, not an external.(I should have) Then deleted the files from the network folder. I'll have to check to see if I can restore that particular folder from a server backup but I don't think it'll still be there.
I'll try deleting the broken links in my library. Just a little frustrated because I pride myself in actually purchasing my music and then I pull a bonehead move and lose most of it.