SE 3rd gen taking too long to set up

Though it is not clear how you are doing the transfer, I have this to say...

If you have already set up the new iPhone then the data created during this period will be lost if you use any of the three processes given below

iPhone to iPhone, there are three ways,

1. Quick Transfer --> Use QuickStart to transfer data to a new iPhone, iPad or iPod ...
2. Restore from iCloud Backup --> Restore or set up your device from an iCloud backup - Apple ...
3. Restore from iTunes backup --> Restore your iPhone, iPad, or iPod touch in iTunes on PC ...

Instead, try transferring data manually. One by One

1. If your old iPhone had iCloud enabled Optimised storage enabled then please do the same on the new device too. Check if you have enough iCloud Storage if you are using another Apple ID.
2. Messages, Media, Voice Memo, Keychain --> Turn on Messages in iCloud & iCloud Photos, Voice Memo, Keychain, and your content is automatically stored in iCloud and is available on all devices that use the same Apple ID including iCloud Drive
3. Contacts --> Contacts are not stored on an iPhone nor on SIM, they are synced to  Google, iCloud, AOL, Exchange, Outlook, Yahoo or any Other. Use the email account that is in your old iPhone.
4. Notes, Reminders, Calendars --> You may sync all these from any of the email accounts that you have on your old iPhone. they can be synced using  Google, iCloud, AOL, Exchange, Outlook, Yahoo or any Other.
5. Any Specific Files --> Upload to Cloud Drives (iCloud Drive, Google Drive, OneDrive, DropBox, Flickr, box, AmazonDrive, Adobe Creative Cloud and Use Files app on your new iPhone too.
6. Apps & Settings --> In any way they have to be redownloaded using the App Store. Even you had used any of the 3 processes mentioned earlier. Apart from the Apps following will also not get transferred.
   1. Content from the iTunes and App Stores or PDFs downloaded directly to Apple Books
   2. Content synced from iTunes, like imported MP3s or CDs, videos, books, and photos
   3. Data already stored in iCloud, like iCloud Photos, iMessages, and text (SMS) and multimedia (MMS) messages
   4. Face ID or Touch ID settings
   5. Apple Pay information and settings
   6. Apple Mail data
   7. Apple Music
7. WhatsApp --> Take a backup of chats and media from within the WhatsApp App. Use the same Phone number as that of your earlier iPhone and activate the account in the new iPhone too. That's it.
8. Shazam Account --> Access your Shazams on all your devices or online - Apple ...