restricted account in messenger

When I had someone as a restricted account before, I could see their messages. It just wouldn't tell them when I was online or if I read their message. For some reason though, now I can't see any of the chats with people I have restricted. I have to unrestrict them in order to look at their chat. It is now showing "end-to-end encryption" like it would if I were to start a secret conversation on messenger.

Any idea on how I can get back to seeing the chats when accounts are restricted?