Protecting ~/Library access from downloaded apps
I know that Mac OS has a permissions model to protect directories like the Document and Desktop directories: Control access to files and folders on Mac - Apple Support.
But I notice that apps seem to have unrestricted access in other directories such as ~/Library. I tested this by opening Firefox and I'm able to freely browse around there, and open up files to view their contents. Doesn't this mean that a rogue app can go into my Chrome or Firefox directory and read all of my saved personal info? What is the correct way of protecting myself when running things like games from the internet?