Are downloading large files from the App Store, Safari browser or terminal affecting overall system performance until they complete?

Probably. kernel_task is a symptom of a problem, not the problem itself. Find the problem and fix it. Read If kernel_task is using a large percentage of your Mac CPU - Apple Support.

A runaway kernel_task can be due to a number of factors, such as:

• a failed or disconnected temperature sensor
• cooling system inefficiency (an inoperative exhaust fan on Macs so equipped, possibly clogged with dust, including the heat sink)
• a worn out, missing, or non-Apple (user-installed) battery on portable Macs
• similar hardware-specific causes including internal hardware additions or modifications, external USB and other connected devices, or the driver software required to use them.

Each of those possible causes is equally important and requires its own investigation.

If you can find no other explanation for its behavior, an SMC Reset is justified: Reset the System Management Controller (SMC) on your Mac - Apple Support.

Downloading files over the Internet should only affect Internet download speed. The system as a whole should not be affected.

To identify potential burdensome processes use Activity Monitor: Use Activity Monitor on your Mac - Apple Support.

Files downloading will result in disk activity which in turn prevents anything else happening until the disk activity stops. This is usually why you may sometimes see the colored beachball for short periods. This is usually intermittent which would not slow down the computer for extended periods like a full download. This assumes you have adequate RAM installed and are not running too many processes concurrently.