You can easily move your Windows files to your new Mac and use them with Mac applications. Macs can open many different kinds of files from your PC, as long as you have appropriate software installed that can interpret them.
Yes. Download & install the HMRC Basic PAYE RTI software for the Mac. Then on the PC backup (using the menu item on the PAYE software) to a removable drive. Take the removable drive to your Mac & restore on the Mac (using the menu item on the PAYE software). Job done - you can now use the HMRC PAYE software on the Mac.