when migrating to a new mac with less storage than your old mac, what is the best way to split up the files between the macintosh hd and other storage?

Some things to ponder over.

1) How much space is used on the old Mac and what size is the new Mac's drive?

It is best to allow plenty of room on the new Mac for migration and growth.

Clean out the old Mac until there is 100GB less than the new Mac's drive.

Consider returning the new Mac and ordering one with a larger drive.

Free up storage space on your Mac - Apple Support

2) Yes you can move your photo and/or music library to an external hard drive.

However you need to use 2 reliable external hard drives for that.

One drive for the libraries and a one to backup the first.

Move your Photos library to save space on your Mac - Apple Support

Change where your music files are stored on Mac - Apple Support

3) How were you planing on Migrating from the old to the new Mac?

Large Network migrations are not reliable and prone to fail.

Ethernet to Ethernet is more reliable but very slow.

Using your Time Machine backup is best.

Transfer to a new Mac with Migration Assistant - Apple Support

Back up your Mac with Time Machine - Apple Support

This eval may be easier for you if you do "About this Mac..." from your Apple menu and select the "Storage" tab. Click the "Manage..." button to get this:

It shows the larger files in groups in the sidebar, but clicking "Reduce Clutter" show individual files sorted by size:

In this shot, I can see five movies I could move to external storage, plus an MS installer I could probably delete.

You're welcome.

IMHO the 2T upgrade to the Mid and Upper level M2 Mac Mini's is worth the extra $600, versus farming out your Libraries to External drives. Buy Mac mini - Apple

Put the lesser used user data on the external drive. But don't fail to back up that data too if it's important to you.

Generally photo, video and sound libraries are easiest to transfer and use, since they tend to be self-contained.