Is the “trusted phone number” inside your Apple ID a security flaw?

Imagine your iPhone gets stolen. Inside this iPhone is your SIM-Card. That SIM-Card is your only SIM-Card and it is enlisted as "trusted phone number" in your Apple Account. The iPhone is the only Apple product you own.

(Note: I believe this setup is not "special" at all, but rather common.)

The attacker now takes out the SIM and puts it in another phone where the PIN/PUK is easily cracked within less than 24h. Enabling the attacker to receive recovery messages/calls.

Apple now provides account recovery based on your enlisted "trusted phone number".

I understand that the phone itself remains safe since it has its own secure password. But the Apple ID with iCloud is at risk.

What can you do to protect yourself against this scenario?

Since it was the only Apple product and SIM you had, is it going to be hard to secure your Apple ID with email and password when your iPhone and SIM goes missing?