If you use File>Export Original, then you will get a copy of the original picture file that has none of your additions.
If you use File>Export nn Photos, and check to keep "titles, keywords, and captions" and "Location," then that file will include all your stuff in IPTC or Exif, the file's metadata. (Well, that seems not to be totally true--hang on...) Most photo editors can read that stuff and, of course, people who have Mac Photos can see it just as you do.
But for others, I've decided to include a PDF file of the pictures that includes the captions, like this:

You get this choosing File>Print (⌘-p) and clicking on "Contact Sheet." You can choose how many columns you want, and therefore the size of the pictures. With the "Captions" option you can choose what to include with the picture, including title, captions, keywords, date, and so on. This works very well.
Now, the weird thing that I just discovered: I just exported a file, and all is as it should be--except the caption in IPTC. The picture had actually had an IPTC caption before I imported it to Photos. In Photos I replaced this original caption with a less generic and more descriptive one. Now the IPTC metadata of the file I exported had my keywords and title that I added in Photos, as expected, but the caption was the old one with no hint of the new one. It also appears this way in Finder's Info window.
So weird.