Sounds like you tried to install an update to macOS High Sierra, you can only install updates if you are running
an earlier version of the OS you are trying to update.
Upgrading to a newer OS needs you to download and install the full installer app which is called
Install macOS xxxxxxx.app. Follow the instructions below.
Which OS you can upgrade to will be governed by which model and year your mac is.
Click on the top left of your screen click on About This Mac to find out the year of your mac.
For macOS High Sierra, macOS Mojave and macOS Catalina,
Open Safari and click on this link,
How to get old versions of macOS – Apple Support
Go to Check Compatibility to see if your mac qualifies for the OS upgrade you want.
Then go to the Download OS section and click on the OS you want to install.
Before upgrading make a full back up of your Mac.
Check that third party apps you use have updates available for the new OS.
https://roaringapps.com
Check that your printers/ scanners etc have updated drivers and software available for the new OS.
Connect your mac to your router via cable rather than rely on WiFi.
Do not use your mac while it is downloading
and do not use any other devices that share the same internet connection.
If the App Store says ‘Open’ then it is detecting a previous download of the install app
go to the Applications folder locate Install macOS ******.app.
Move it to the Trash, empty the Trash.
Restart, click on the link again, the App Store should say ‘Get' or 'Download'.
If you want to or are able to install Big Sur, open Safari and click on the link below.
How to upgrade to macOS Big Sur - Apple Support
If you are considering upgrading to Catalina or Big Sur be warned they do not support 32bit apps.
Your Mac is probably running 32bit apps and these will need to be updated to 64bit apps,
some updates may be free, some may be paid. Use this app to discover which apps you have
are 32bit, https://www.stclairsoft.com/Go64/ then contact the developers of the software to see
if they are offering 64bit updates.