I can't disable voice control.

In the Accessibility > Voice Control panel, see Commands… There is a selection among the commands named Turn off voice Control, that annoyingly only stays selected during the current Voice Control/Dictation session — if you think to select it before enabling Voice Control.

Once the first paragraph is done, you can speak Turn off voice Control and the Voice Control and the onscreen Dictation panel are switched off. The next time you want to use Voice Control, you must repeat the first paragraph to enable the spoken command again.

You can also use an AppleScript to toggle Voice Control/Dictation on and off each time you run it. The folowing script works with macOS 11.2.3 to do just that. If Voice Control is on, the script will turn it off. And if you want to start Voice Control and have the Dictation panel appear, you run the script again.

Copy/paste the entire script into Apple's Script Editor (Dock > Launchpad > Other > Script Editor). Click the compile button. Save it twice:

1. Save…
   1. File Format: text
   2. Name: toggle_voicecontrol.applescript
   3. Save to your Documents folder
2. Option + File menu > Save As…
   1. File format: Application
   2. Name: toggle_voicecontrol.app
   3. Options: leave these alone
   4. Save to your Desktop

When you double-click the toggle_voicecontrol application now on your Desktop it will turn Voice Control on or off depending on Voice Control's current state.

-- toggle_voicecontrol.applescript
-- Future releases of the operating system may break this script
-- How to launch and quit Catalina Voice Control using GUI scripting.
-- Update: tested with macOS 11.2.3 - 2021-03-19
-- VikingOSX, 2019-12-7, Apple Support Communities, No warranties expressed or implied.

set osver to (do shell script "sw_vers -productVersion | cut -d'.' -f1-2")

tell application "System Preferences"
	launch
	reveal anchor "Dictation" of pane id "com.apple.preference.universalaccess"
end tell

tell application "System Events"
	tell application process "System Preferences"
		set frontmost to true
		delay 1
		if osver ≥ "10.15" then
			tell window "Accessibility"
				delay 0.5
				-- where checkbox text is not in English
				-- click checkbox 2 of group 1 of it
				click checkbox "Enable Voice Control" of group 1 of it
				click button 2 of it -- red traffic light on the opened panel
			end tell
		end if
	end tell
end tell

One side effect of using this GUI AppleScript is that it turns on menu overlay numbering when Voice Control is active — even though that overlay feature remains set to None in the Accessibility > Voice Control panel. Uncertain of how to fix that annoyance.

Try turning off Dictation under the Keyboard Preferences.

Go System Preference >> Siri and Disable Voice Control

Try Command + F5

No problem!