iPhone X side button delay

Put your device in recovery mode and set it up again

1. Make sure that you're using the latest version of iTunes.
2. If iTunes is already open, close it. Connect your device to your computer and open iTunes. If you don't have a computer, borrow one from a friend or go to an Apple Retail Store or Apple Authorized Service Provider for help.
3. While your device is connected, force restart it with these steps, but don't release the buttons when you see the Apple logo, wait until the connect to iTunes screen appears:
   • On an iPhone X, iPhone 8, or iPhone 8 Plus: Press and quickly release the Volume Up button. Press and quickly release the Volume Down button. Then, press and hold the Side button until you see the connect to iTunes screen.
   • On an iPhone 7 or iPhone 7 Plus: Press and hold the Side and Volume Down buttons at the same time. Keep holding them until you see connect to iTunes screen.
   • On an iPhone 6s and earlier, iPad, or iPod touch: Press and hold both the Home and the Top (or Side) buttons at the same time. Keep holding them until you see the connect to iTunes screen.
     
4. When you see the option to Restore or Update, choose Update. iTunes will try to reinstall iOS without erasing your data. Wait while iTunes downloads the software for your device. If the download takes more than 15 minutes and your device exits the connect to iTunes screen, let the download finish, then repeat step 3.
   
5. After the Update or Restore completes, set up your device.

I had the same issue, its actually Apple Pay that is causing the issue. I tried a restore and it didn't help. Turn off double click side button in the wallet app and you'll notice right away that the phone is a lot faster when you click the side button. Lemme know if that helps. It worked for me and I haven't had that issue since, and I know what you mean man that issue was really annoying.

Well I never use it but you can open the wallet app to pay, you can add it to the control center, or I’ve heard all you gotta do is put the phone near the reader and it comes on automatically, then you just double click to pay and authenticate using your face.