Removing files from your iMovie library likely will disable many if not most of your projects.
Instead, you can move your 500 GB iMovie library to a separate drive that is formatted Mac OS Extended (Journaled) or APFS. First, select the library's name in the iMovie side bar, and do a File/Consolidate library media. Then when you transfer the library all of the media in the library will go with it and off of your internal drive.
Then create a new iMovie library on your internal drive. Populate it with whatever media that you want, or store the media in a finder folder on your internal drive, or on an external drive, for later import into iMovie. You can also access any media that you have stored in the Photos app library by selecting the Photos item in the iMovie sidebar.
-- Rich