IMO the most readily evident potentially “negative impact” scenario:
You are traveling somewhere out of town carrying ONLY a single Trusted Device; presumably your iPhone.
If that phone is lost, damaged, or stolen; it’s fairly easy to purchase a replacement device.
BUT without one of your pre-registered, NFC-capable hardware security keys in-hand - you will NOT be able to add the replacement phone to your Apple Account or restore from an Automatic iCloud Backup.
So … thinking-thru — and actually rehearsing — the various recovery scenarios pertinent to your own particular situation is of paramount
importance for one devides to optionally use
hardware security keys.