Hi Yavno,
Sadly, no. A whitelist feature, in Safari, would be a VERY welcome addition. As it stands, Remove and Remove All are the only two options. Hand-picking those you want to delete, while preserving those you want to keep, is painful at best. 🤨
With that said, I am not sure it is an oversight. Apple may not intend for users to actively delete these as frequently as I, and it sounds like you, are inclined to do. I notice that doing so skews the metrics that Safari reports in regards to cookies & trackers blocked. Also, I have noticed that after having done Remove All to wipe the list clean, I can return at some later point to find websites back on the list that I have, without doubt, not ever visited again. It would seem that the How and Where behind Safari's storage and retrieval of this information is more complex than that of other browsers, which may well make a whitelist function difficult to implement.