Yes, you can safely remove the majority of them. All active fonts take up RAM. The less fonts that are active, the more RAM that is available for other uses.
See my article, Font Management in OS X, to find which fonts can be removed for each major release of OS X (since 10.3).
I practice what I preach. The lists you see are all I ever have permanently active. Many folks are unaware that there is a darn good reason for doing this, depending on your line of work. I do prepress work all day. The less fonts I have open, the better. Especially OS supplied fonts that would conflict with customer supplied Type 1 PostScript or other font versions that must be used with a project.
I may receive some form of compensation, financial or otherwise, from my recommendation or link.