MacBook Air asking for CAC pin even when I don't have the CAC reader connected/CAC inserted

Subject line sums it up well. I sometimes use a CAC reader with CAC on my MacBook Air (M1, 2020) to access my secure work website (military). But sometimes, even when my CAC reader isn't connected, and when I open up my Mac to login, it asks for my PIN (associated with my CAC reader/CAC card), and won't let me log on unless I plug in my CAC reader, insert my CAC card, and put in the pin. This is a pain for obvious reasons. Anyone else encounter this problem? Any fixes? I have no problem with the Mac asking for my pin when my CAC reader and CAC are actively connected, but it makes no sense for it to require my pin when I'm not even using the CAC reader. Thanks!