You can create single-level "subsets" in iBook using the "Collections" feature. At the top of the iBooks window is a row of buttons. Click (tap) the second button, Collections. That will bring up a sidebar on the left listing the various collections you have. At the moment, they're probably just the default collections. Click (tap) the "+" button in the lower left corner of the sidebar to add a new collection, then go to your default collection containing your pdf ("PDFs", natch) and drag them individually (or command-click (tap) to select several) to your new collection.
iBooks can't create nested collections, however, so if you want to organize your files in a hierarchal set of nested folders, you'll have to use another app. Lots of good ones out there, some free, some not. Free apps include Readdle's excellent "Documents" app that allows nested folders as well as annotation and even a built-in web browser for finding and downloading pdfs. The app syncs with iCloud, Dropbox, and other cloud services and lets you browse files on your Mac. It also keeps your files synced across your iOS devices (there are iPhone and iPad versions of the app), which can be VERY handy. Readdle also makes another app called "PDF Expert" that offers other features you may need (but no web browser). I have both, but can't decide between the two and find myself switching back and forth for no real reason at all. Start with Documents, but check out PDF Expert. I think you'll find both will do a better job of organizing and accessing your pdfs than iBooks.