Perhaps there should be a calendar named “Observed US Holidays” which is what I believe the OP and many others are looking for.
It isn't possible, since there are so many different ethnic and religious organizations in this country. Every holiday on the default calendar is celebrated by someone.
I get what you mean by days workers get time off for, but that isn't even close to universal, either. Where my younger son works, the one and only "normal" holiday the workers get as a paid holiday is Christmas day. Any other standard holiday you have to use vacation time for. And the whole warehouse can't get an okay on their request since product still has to be distributed to the stores 24/7.
Okay, what about the rest? Who gets President's Day, Martin Luther King Jr. Day, Columbus Day or Veteran's Day off? Pretty much only government employees. For virtually any other company, you work those days.
What about Ground Hog's day or Valentine's Day? They're official U.S. holidays, but I don't know of anyone who gets them as a paid, or even unpaid day off of work. Not even government employees.
There are no truly fixed holidays that everyone gets off of work. You can take any holiday of the year and note how many people do not get the day off paid. Such as hospitals, police and fire departments, gas stations, department stores (they love being open on holidays), and many others.
Long ago in a print shop I worked for, one of the office staff was Jewish. Christmas meant nothing to her, but one of the bigger ones in her faith, Rosh Hashanah, did. So they would give her that day off as a paid holiday and she'd come in to work on Christmas day. She said it felt odd to be the only person in the building, but it was also a day she could get a lot of CSR work done without being pulled into other projects.