Yes, you need to start over and format the drive MacOS Extended (Journaled) or APFS . With a FAT32 format you may encounter problems with the filenames of the internal files in the library and lose images. Prepare the thumbs drive as described here: Move your Photos library to save space on your Mac - Apple Support
Once the iPhoto Library is on the thumbs drive, connected to your new Mac, you can import the images and videos into a Photos Library. Open the Photos Library in Photos, then use "File > Import" and select the iPhoto Library. Photos will show you all new photos, that are not yet in your Photos Library and you can then import all new photos (and videos).
It is up to you to combine all libraries into one or to keep them separate. Photos you will need to use together for a project should go into the same Photos Library. Otherwise it helps to keep the Photos Library small. Photos is doing a lot of background processing to analyze the photos to recognize the scenes, recognize faces, pets, objects, read the text on the photos, so we can ask Siri to find our photos for us. Photos is also using artificial intelligence to suggest photos for sharing or to create memory movies. This can take weeks, if the library is large.
There is one problem - your new Mac mini probably came with macOS 13 Ventura. On this system version you can no longer convert iPhoto Libraries to Photos Libraries. You can only import them into an existing library. That will save the photos and videos, but not the album, folders, events. The structure of the library will be lost. if you still have a Mac with an older system version, you can convert your iPhoto Libraries there to Photos Libraries and save the structure of the libraries.
The older Photos Libraries can simply be opened in Photos on your new Mac.
