Referenced fils are not saving space - they are just a an added complication and can be causing a lot of trouble.
A Photos Library is storing the original image files plus some working copies to make Photos faster (thumbnails for browsing, previews of the edited versions, faces thumbnails, also database files to manage the structure of the library (albums, smart albums, folders, projects) and the metadata and adjustments of the photos.
The storage for the originals will be the same, whether the originals are stored in the library or referenced outside the library. And the storage for the working copies will also be the same. Usually the working copies will need roughly 205 to 30% of the size of the originals. If the library size doubles, you may have been syncing your library to an iPhone, iPad, or Apple TV and photos may have created a cache of the synced photos. These caches can be quite large.
Or are you using iCloud Photos for syncing with the option "Optimize Mac Storage"? Then the size of your library may have increased, if original image files have been downloaded from iCloud.