The main headphone jack on a Mac Pro 65lb tower is for a 3.5mm plug (not the 2.5mm used for many earbuds).
The first thing to do is check in System preferences > Sound > Output ...
... and make sure the settings there are appropriate.
Headphone jacks have a mechanical component, and as such they can be fussy. Sometimes it take inserting, wiggling, and removing the plug a few times to a dozen times to get it to work correctly, then hopefully it will continue to work with one insertion after that.