Well, yes you can move them off - preferably to a second drive - if you have a lot of GB's worth. If it's not that many, then don't worry about it. If you have the libraries on a second drive, then yes, you do want to make sure that you also have a backup of those. I believe Time Machine can handle backing up an external; however, I don't use it, so make sure you check that (I prefer working with bootable clones).
I'd consider moving them if you have a lot and if you're into editing video/making movies as you do need as much empty hard drive space available for reading/writing while editing, rendering, and burning as you have in project files. I regularly have 150 - 200 GB of files in an HD project, so I need to make sure I have another 200+ available. While in a project, my hard drive space used easily quadruples until I'm done and the temp files are archived on an external or deleted.
As mentioned, you should have 2 - 3 times for TM (if you are thinking of filling up your 1 TB drive - see last paragraph).