The encryption key is stored on the disk. Both Disk Utility and FileVault. The Encryption key itself is encrypted by the passphrase for Disk Utility and by the user login password(s) for FileVault.
The 2 stage encryption is done this way so you can change your password, or the passphrase and just need to decrypt the encryption key and re-encryption with the new password or passphrase, which can be very quick. If your password or the passphrase was the actual encryption key, you would need to decrypt the entire disk and re-encrypt it if you wanted to change your password or the passphrase. Also, the encryption key is much larger and more secure than most people are willing to make their passwords and passphrases. Although the better you make your password or passphrase, the more secure the encryption of the key itself will be.
Also encrypting the key using your password, allows FileVault to store multiple copies of the encryption key, each encrypted by a different user's password, so more than one person can unlock the disk.
Any time you expose your data to someone else, you MUST trust the other person. If they have hacked their system so it can read the encryption key from RAM, or they have hacked it so they capture your password or passphrase as you are entering it, then they can unlock your disk anytime they want.
Also once your disk is unlocked, the device drivers will let ANY program running on that computer, free access to the data on your disk. Remember, no matter how you set ownership and permission on files, the owner of the computer can become 'root' and have full unrestricted access to all parts of that disk.
The owner of that computer can also modify things on that disk, including programs that you may have stored there to substitute there own.
And it does not need to be the owner themselves. If the owner has malware on their system, the malware could be accessing your disk. If your data is so sensitive that you might be targeted, then a bad actor could infect the owner's computer with special malware aimed at you, if they know you regular expose your data to that owner's computer.
So "Do You Trust That Computer And Its Owner?"
The purpose of whole disk encryption is to prevent someone that gets your disk and NOT your password/passphrase from accessing your disk. If you give them the password or passphrase (even if indirectly by typing it yourself), whole disk encryption is now defeated.