What does turning on library sync do to music files on your Mac?

See Use Sync Library to access your music library across your devices - Apple Support. In principle nothing at all should happen to the existing music on your computer, although the library may get the addition of any content or playlists that currently exist in your iCloud Music Library if you've made use of Apple Music on another computer, library, or device. Your local content should be examined and either matched to items in the iTunes Store, or uploaded where no match is found, but the local files remain where they are unless you actively choose to use the remove download option. Always best to backup your library before making major changes regardless of how things are supposed to work.

Alternatively you can create a separate library for Apple Music, enable Sync Library, and then selectively introduce copies of your own content that isn't readily available in the Apple Music catalog, leaving your original library completely untouched.

tt2