Apple Pay on your Mac mini M4 should not normally require your Mac login password every time you make an online purchase.
Since the Mac mini does not have built-in Touch ID, Apple Pay is usually approved using either an Apple Touch ID-enabled keyboard, your Apple iWatch, or your iPhone if "Allow Payments on Mac" is enabled.
If your Mac is asking for your Mac password instead, it usually means it is not able to communicate with your Apple Touch ID-enabled keyboard, Apple Watch, or iPhone for payment approval.
In that situation, macOS falls back to using your Mac password as the authentication method.
The first thing to check is that your Apple Touch ID-enabled keyboard is paired with your Mac and that Touch ID is working correctly for unlocking the Mac and other authentication requests.
Also confirm that your iPhone has "Allow Payments on Mac" enabled in Settings > Wallet & Apple Pay.
Verify that your Apple iWatch is unlocked, has a passcode enabled, is using the same Apple Account as your Mac, and that your payment cards are available in Wallet.
Another useful clue is whether your Apple iWatch can unlock your Mac.
If that feature is also not working, the problem is likely related to communication between the iWatch and the Mac rather than Apple Pay itself.
Bluetooth, Wi-Fi, and two-factor authentication all need to be be enabled for these features to work correctly.
Apple Intelligence may have been used for the wording; the technical content reflects my own experience