I have found out more. Apparently the new version of iBooks requires ePub3 using OCF3; however it is, the new version uses a different kind of file than the old one. The good news is you can drag all your old Calibre-created ePub files into iBooks and it will convert them to the new format, and they will be available and useful. The bad news is that they will take up a lot more space.
With the old version of ePub, the file was a folder that contained the book in HTML format, in chapters if necessary, and some additional information like the CSS document and the cover. That folder was compressed with ZIP. If you knew what you were doing, you could set up the folder, zip it, and change the .zip extension to .ePub; I made a few that way. Just as a test, I took one of the old ePub books, changed the extension to .zip, expanded it with Stuffit Expander, and opened the file. Then I checked the contents of the folder. It included a stylesheet.css file, a content.opf file (that's used by Calibre), a toc.ncx file (table of contents added by Calibre), one .htm file for each chapter, a JPEG version of the cover, a title page.xhtml file (which only contained the cover JPEG), and a mime type document.
When I look at a new version it is different. It is now a Package. When I open package contents I see almost exactly the same contents as the older ePub unzipped (plus an iTunesMetadata.plist file...which shows that iBooks will eventually use "Genre" while iTunes used "Category"...) But in most cases the new version is more then double the size of the old version. Sometimes there's more structure. And files that have DRM have encrypted HTML files.
I spent a lot of time on the phone with Tech Support and was eventually told that I should just get another reader, and it wouldn't have any trouble with the iBooks I had bought on the iBookStore. As it happens, no other reader I tried can even access an ePub through the package that iBooks puts around it. I did really like the Marvin reader; unfortunately, one of its best features--that it can have its bookshelf organized by Calibre--is currently broken. I'm still steamed enough about being treated badly by Tech Support that I am looking for a way to make Marvin my new standard eBook reader, but unfortunately I have over 200 books that I bought with iBooks, and no way to access them using Marvin. I guess I'm stuck with iBooks.