No, it is not expected to lose access or to slow down access to external drives in Catalina.
I am guessing there was a serious miscommunication there.
Catalina has more strict controls of access to files, but once you give an application the requested permission, it should not be any slower than before. An application may be able to do something, or not be. That is it.
How is that drive formatted? Is this a Time Machine drive? If so, you should not be going through the contents in the Finder, you should use the Time Machine interface.
If not, then please tell us how the drive is formatted. If possible, post a screenshot of the Get Info, showing the format and permissions (do not show any private information).