How do I share my desktop folder?

I have enabled SMB File Sharing on my M1 MacBook Pro running up to date Monterey.

I can connect to my Mac on windows and logging in with user name and password I am able to see my User folder contents.

I can access most of the directory contents except for the privileged type folders such as Desktop/Downloads which I get a "you don't have access" error from windows.

How can allow access to these folders via file sharing.

I have tried adding the Desktop folder specifically as a shared folder in File Sharing and given permissions, but this does not help.

It feels like I need to add "File Sharing" into the security & privacy settings of files and folders.

Any suggestions?