Which system version did you use, when you created the Photos Library, that you no longer can open in Photos on High Sierra?
The Photos.app is a part of the system and can only be updated by system updates or upgrades.
Thus, the Photos.app can only open Photos Libraries that have been created or updated with the system version currently installed on your Mac or older Photos Libraries.
Directly after the upgrade to High Sierra there may still have been an alias to an older version of Photos on the Desktop or an outdated icon in the Dock. Try to launch the Photos.app directly from the folder Applications and avoid all Dock icons or alias.
- Select Photos in your Applications folder.
- Hold down the options key ⌥ and keep holding it down until the Library Chooser dialog appears. Release the options key.
- Select your Photos Library from the list of libraries and click "Select Library".
Can Photos open the library this way?