What happens if you click the X on the top right, will itunes backsup with encryption? or not?

Typically that would cancel the current operation, with no decision being made. I would expect you to get that prompt again the next time iTunes wanted to start backing up. When choosing to encrypt backups there should be a second dialog where you enter the encryption password, so it would odd if encryption were somehow enabled without that step.

See Disable iOS backup encryption - Apple Community for some further background.

tt2

1. Yes. I left out the word "be" after "would" in error.
2. To be fair it would also be odd if backup commenced, with or without encryption if you simply close out the dialog. The convention is that this cancels or aborts any proposed activity. But it is just that, a convention.

To be clear I'm not currently in a position to test out this scenario, and it isn't anything I've explicitly tested in the past. I assume you're asking the question because something unexpected happened to you when faced with this dialog.

tt2