Yubikey and trusted phone number, do you need both?

Hello, I just added several YubiKeys to my Apple ID. I am wondering if by having the hardware keys registered to my account, is it unwise to also have the trusted phone number active on the account. My concern is the YubiKeys can be defeated, because the trusted phone number can be SIM hijacked and used to access the account. I am new to using YubiKey and not sure how best to protect myself against hacking. Any feedback is appreciated. Thank you.

‘regards,

‘Bob