Screenshot glitch in OS-X Catalina:

Check to see if there's this button in the menu bar. It's be the farthest left icon on the right side:

It appears when I'm doing a recording of the screen or window. Clicking on it stops the recording and clears the window. See if that's what you're experiencing.

I finally figured it out. For whatever reason, the Screencapture app defaulted to a video capture rather than a frame capture. What was happening as described - framed area remained white while the rest was greyed out, the app would not respond to commands - was the app was recording a video. I never did discover how to end the recording, and the app may have exceded capacity and frozen. I had to restart the computer to reset everything. Changing the default command in the small strip underneath fixed the issue, returning Screencapture's default to an actual single screen or frame capture. I still don't know how one would actually attempt a running video capture, and that may be beyond my high-end MBP, but I don't care. I'll never use that function. All the talk of preference files, etc, were off target. It was a simple mistake of what default command happened on a 'return' button.

Oh yes, I've seen it. It happens to me a lot on a Mac mini running Mojave! It happens so often, i just use the old CMD + SHIFT + 3 or 4. Although I think Screen Capture works correctly on my MacBook Air that came with Mojave.

The capture controls are supposed to go away if you press ESC, but this (often) doesn't seem to work. I've never found a fix that solves this, but to get rid of the gray screen open Activity Monitor and force quit "screencapture".

Log into another user account and see if the problem persists there in Photos. (This tells us if the problem is limited to your user account or is system wide).

Then boot into Safe Mode (Use Safe Mode to isoloate issues with your Mac) and try from there. Reboot normally and try again.

NOTE: Safe Mode boot can take up to 10 minutes as it's doing some system cache cleaning, volume verifying and directory repairing. 

'command-shift-5' produced nothing. Using the drop down menu, I unchecked 'remember last selection', then tried screenshot, and got the typical freeze. Screenshot could not be closed, no screenshot menu could be produced, the greyed out area not surrounded by screenshot outline remained, and even 'force quit' showed nothing for screenshot. The only way to escape this is to restart.

Any other suggestions? I already deleted screenshot preferences and got same behavior.

Press "command shift 5, click on Options, uncheck "Remember Last Selection".

Again, if Screen Capture isn't working correctly, open Activity Monitor and force quit "screencapture".

Safe Mode might clear it up for you for the long term, but hasn't helped me.

Old Toad explained that 4 posts above and how to stop the recording.

This didn't work. It still left the frame of the screenshot.

It doesn't help either because I want it to remember the last selection. Otherwise I would use Command Shift 4.

Okay, this details the issue and the solution to a tee. Thanks JCampbell for fixing my MBP!!!

What’s the shortcut to take screenshot