How do I use a 2017 MacBook Pro as an extended display for a 2021 MacBook Pro?

Note that while there is no way to set up the old MBP as an AirPlay display using only the support built into macOS, there may be third-party applicationsfor the old Mac that would let you AirPlay to it. These applications are likely to be based on the older AirPlay v1 protocol, and to limit resolution to 1920x1080 at best.

It appears that you have been using AirPlay to Mac.

Continuity features and requirements for Apple devices - Apple Support

Your M1 MacBook Pro is recent enough to meet the requirements for being an AirPlay Receiver for AirPlay to Mac. Your 2017 MacBook Pro is not. That is why you can't use the old MBP as an AirPlay display for the new one.

Why does it work in the reverse direction? According to Apple's notes,

"You can AirPlay to Mac at lower video resolutions from older iPhone, iPad, and Mac models and operating systems if the AirPlay Receiver setting is turned on and the adjacent "Allow AirPlay for" setting is configured to allow AirPlay either for everyone or for anyone on the same network."

This makes sense if you realize that Macs have been able to AirPlay to Apple TV set-top boxes for a very, very long time. Your older MBP can AirPlay to Apple TV set-top boxes (even if it can't be an AirPlay Receiver itself) – and, in this case, your newer MBP was in effect pretending to be an Apple TV set-top box.