iOS Apple Mail badge-notification doesn’t update

I have an issue with the badge on my Apple Mail in iOS.

Examples:

• The badge on mail displays “4 unread emails” - I open the app, read two mails and move on to something else. The badge still displays “4 unread emails”. Even though the “All inboxes” only has 1 unread.

• The badge is not present, but when I open the app there’s 7 new messages.

Sometimes it helps to update the inboxes (by swiping down) but not always.

Force quitting the mail app doesn’t make a difference.

I use a XsMax with IOS 13.5