Is it best to NOT migrate data from an older IMAC to a new IMAC to start fresh?

My IMAC is 7 years old and nearing legacy status. I am going to buy a new 24 inch IMAC to replace it. If I use migration assistant I can move everything over seamlessly and without any hassle or additional setup. However, my older IMAC has seen many apps come and go and I have concern that the HD is riddled with bits and pieces of code that were left behind. Old Caches, etc. Should I bite the bullet and NOT migrate all my data from my old IMAC and start over? I know I can download files from ICloud and I would intend to do that....including photo library, documents and desktop. But should I reinstall all apps? Or start over? Many thanks!