I am pretty confident the issue appears with folders that have sub-icons, like the cloud sync state icon, alias icon, or other icons that appear in a corner of the folder icon. I've only seen it happen with cloud folders and aliases.
This explains why disabling the dropbox finder extension fixes the issue for dropbox folders; it removes the cloud icons.
Is there a way to disable all these icons? That would likely fix the issue.
Anyways, while it isn't a fatal bug, it's visually ugly, and the reason I chose MacOS is to avoid bugs like this. Tahoe has brought many of them, and I hope they'll be resolved soon. Can't believe this bug has lived for so many months...