How can I set up new iPhone SE without using iCloud?

You need not use iCloud at all. Skip the step of creating iCloud and syncing iCloud.

• In step 2 choose: "Setup Manually"
• In step 5 choose "Dont's Transfer Apps and Data"
• Skip Steps 8 & 9, this can be set up later also.

Set up your iPhone, iPad, or iPod touch - Apple Support (IN)

If you want to transfer data from a previous device without using iCloud just back up the old device to your computer using iTunes (or Finder if you have a Mac with MacOS Catalina or later), then restore the backup to your new phone. The instructions are here→Transfer data from your previous iOS or iPadOS device to your new iPhone, iPad, or iPod touch - Apple Support.

To transfer all data you need to make the backup encrypted, but don’t forget the backup password, because there is no way to recover it.

Note that you still need to log in to your Apple ID on the phone (which is not the same as using iCloud) if you want to use Find My.

Using a Mac, or PC with iTunes was mentioned, and that's how I do it.

But if you have a newer iPhone with the old data, you might be able to use Quick Start. But you only get one chance at Quick Start with an iPhone at factory settings unless it's restored to factory settings later. It does the transfer via Bluetooth and can be somewhat slow. It's also important to keep both from running out of battery power (especially with active Bluetooth transmission), so both should be connected to power.

Use Quick Start to transfer data to a new iPhone, iPad, or iPod touch - Apple Support