Why can’t I download a free app without a payment method

You may need to have a valid payment method on file, even if the app you want to download is free.

You can have a credit card / debit card on file and an Apple Account balance.

If you have set your Apple Account balance as your default payment method, your default payment method will be used first by Apple when you need to make a payment.

Some purchases might require you to have a credit card on file, even if you use your Apple Account balance to make the purchase.

1.

You can download free apps, but you may need to have a valid payment method on file, even if the app you want to download is free. 

As long as you only download free apps, it costs nothing.

Then your payment method remains intact.

If you are going to download a free app, there is a possibility to purchase the 'better' version, for example from a free version to pro version.

But that is not mandatory.

Use your Apple Account balance as default payment, then it is up to you whether you use it or not.

Free apps has also In-app pruchase.

Your payment method is actually a kind of deposit, if you buy something, it will be charged.

2.

If you don't have a credit card or don't want to use a credit card, you can also use your Apple Account balance as payment method.

See -> What you can buy with your Apple Gift Card or Apple Account balance

What you can buy with your Apple Gift Card or Apple Account balance – Apple Support (UK)

⤵️

• You can't use your Apple Account balance to send gifts or gift cards.
• If you use Family Sharing, your family members can't spend your Apple Account balance. Funds that you add to your Apple Account balance are only available to you. Find out more about how purchases are billed to your Apple Account balance.
• Some purchases might require you to have a credit card on file, even if you use your Apple Account balance to make the purchase.

See also -> How to redeem your Apple Gift Card or App Store & iTunes gift card

How to redeem your Apple Gift Card or App Store & iTunes gift card – Apple Support (UK)

When redeeming a gift card, the amount will be credited to your (1) Apple Account balance and (2) you need to set your Apple Account balance as your default payment.

(1)

See -> Check your Apple Account balance

Check your Apple Account balance – Apple Support (UK)

(2)

See -> Add a payment method to your Apple ID

Add a payment method to your Apple ID – Apple Support (UK)

If you already use your Apple Account balance, the Apple Gift Card or App Store & iTunes gift card is automatically added to your Apple Account balance.

If you update your payment method or add funds to your Apple Account balance, It would take about 1 to 3 business days in order to process.

If you're still unable to update your payment method, Contact Apple Support.

See also -> Payment methods you can use with your Apple ID

Payment methods you can use with your Apple ID – Apple Support (UK)

fyi: Not all credit & debit cards seem to work with App Store / iTunes Store, try a different one, or preferably a credit card. If your card is not accepted as a paymenf method, you need to contact your bank.

-----

If you have a valid payment method on file, instead of buying or order an App Store & iTunes gift card, you can Add money to your Apple Account balance anytime, no gift card needed, see -> Apple Gift Card - Apple (UK)

See also this Apple article:  If you can't redeem your Apple Gift Card or App Store & iTunes Gift Card

If you can't redeem your Apple Gift Card or App Store & iTunes Gift Card - Apple Support

Beware of -> About Gift Card Scams -> About Gift Card Scams - Official Apple Support 

If you still need help, get more help and assistant from Apple directly, see Contact Apple for support and service -> Contact Apple for support and service – Apple Support (UK) 

The short answer is "Because that's the way Apple set it up.".

(Raicya's answer was excellent and very comprehensive).