I suspect this is being caused by embedded cover art in the music files.
I reproduced the behaviour you're seeing in a folder containing 138 folders with a total of 4,202 music files – a mix of mp3 and m4a format – many of which have embedded cover art.
In list view if I expand them all (with command+a then command+right-arrow) they will gradually all open after a few seconds but when I collapse them (with command+left-arrow) Finder hangs.
I took a sample of the non-responsive process which has this:
Process: Finder [26186] [unique pid 224967]
UUID: CCBC859B-DCAE-366E-9456-CE2C7DC1B5A8
App Version: 11.1
Build Version: 1350.2.10
Path: /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder
Architecture: x86_64
Parent: launchd [1]
UID: 501
Footprint: 218.12 MB
Time Since Fork: 502s
Num samples: 794 (1-794)
CPU Time: 0.005s (14.5M cycles, 3.8M instructions, 3.81c/i)
Note: Unresponsive for 260 seconds before sampling
Dispatch Thread Soft Limit Reached: 64 (too many dispatch threads blocked in synchronous operations)
Dispatch Thread Hard Limit Reached: 512 (too many dispatch threads blocked in synchronous operations)
Those final thread limit warnings are followed by 448 copies of the following (this is the last example):
Thread 0x45c70f DispatchQueue "TIconFetcher Group Queue: 35 nodes"(1759) 794 samples (1-794) priority 37 (base 37) cpu time <0.001s (345.6K cycles, 64.8K instructions, 5.33c/i)
794 start_wqthread + 15 (libsystem_pthread.dylib + 9319) [0x7fff20336467]
794 _pthread_wqthread + 244 (libsystem_pthread.dylib + 13395) [0x7fff20337453]
794 _dispatch_worker_thread2 + 92 (libdispatch.dylib + 77752) [0x7fff2019efb8]
794 _dispatch_root_queue_drain + 326 (libdispatch.dylib + 75863) [0x7fff2019e857]
794 _dispatch_async_redirect_invoke + 713 (libdispatch.dylib + 22632) [0x7fff20191868]
794 _dispatch_continuation_pop + 543 (libdispatch.dylib + 25101) [0x7fff2019220d]
794 _dispatch_client_callout + 8 (libdispatch.dylib + 14279) [0x7fff2018f7c7]
794 _dispatch_call_block_and_release + 12 (libdispatch.dylib + 9693) [0x7fff2018e5dd]
794 ??? (Finder + 2694486) [0x109167d56]
794 ??? (Finder + 518274) [0x108f54882]
794 usleep + 53 (libsystem_c.dylib + 494476) [0x7fff20285b8c]
794 __semwait_signal + 10 (libsystem_kernel.dylib + 14262) [0x7fff203087b6]
*794 semaphore_wait_continue + 0 (kernel + 885280) [0xffffff80002e8220]
I'd take a punt that 'TIconFetcher' is responsible for loading the cover art images and showing them as icons and it's overloaded as the folders open, then if you also ask it to stop and figure out which ones to not worry about as the folders are closed it all screams to a halt until the queue sorts out which way is up..
Interestingly with the folder copied to a Samba share I get very slow performance but it does NOT hang Finder and a grey Loading spinner I haven't seen before appears on the window. Perhaps the latency of the network gives Finder enough time to juggle the requests in a more reasonable way?
To test this I duplicated the folder and removed all of the embedded album art from the files (which resulted in a drop in size from 143GB to 87GB!) which resulted in fairly quick performance expanding the folders and a slight hang (a few seconds of beachball) when collapsing them.
Chunking through 56GB of image data to get tiny thumbnails for thousands of files in real-time is a clearly a bit of an ask but some more graceful feedback, like the loading spinner on the network example, would be nice to have.