Before actually using that feature, users will be asked to authenticate by repeating an on-screen code, to exclude unauthorized users. It is not thought to be a security thread, as passwords will not be shared without access to the TV screen, and only selected credentials will be shared to that one device, using an encrypted bluetooth connection.
Enter text on your Apple TV - Apple Support
To turn off this feature on your iPhone, iPad, or iPod touch, go to Settings﹥Notifications﹥Apple TV Remote Keyboard.
The feature is intended to be used ad-hoc, without pairing, e.g. even when visiting friends. Select a combination of username and password from the iCloud Keychain on the iPhone, to be used for a selected service on the Apple TV. Then confirm using the authentication method (Face ID, Touch ID, or device passcode). Some Apple ID entries may ask for two-factor authentication additionally.
As bluetooth through reinforced walls is very limited, the Apple TV user (tvOS 12 or later) will be very nearby.
Unfortunately, I am not aware of any setting on Apple TV to prevent the pop-up notification on unpaired phones, e.g. to ask the Apple TV user to change their Settings.