The operating system will install Python 2.7.16 into System locations. Leave this alone, though note that Python 2.7 is end-of-life at Python.org. Perhaps as early as Big Sur, Apple has vowed to remove pre-installed Python, Ruby, and Perl scripting solutions from the operating system installation. When those changes take effect, Apple expects users to install their own solutions.
There is no need for a Python library in your local ~/Library, so purge that.
The /Library/Frameworks/Python.framework/Versions/3.8 is the result of using the python.org 3.8.n installer, and all of the binaries from this are linked into /user/local/bin/python3, idle3, etc. You can default access these by manipulating your PATH order.
Yes, the Xcode 11.n and command-line tools for Xcode 11.n installations now toss a reduced Python3 installation into System areas, and as long as the development tools are installed, it remains. I simply pretend that this cruft does not exist.
For the Python.org installation, if you already have the following PATH statement in your Bash ~/.bash_profile as:
export PATH=".:/usr/local/bin:$PATH"
the Python.org installer will later add the following to it:
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
with the PATH now set, when you interactively run python3, idle3, or pip3, you will be invoking only the python.org tools, and not Apple's /usr/bin/python3. Note, that with Zsh, that the python.org installer will not update the ~/.zshrc file, and you have to copy/paste from the Bash dot files.
Also, with #!/usr/bin/env python3 at the beginning of a Python script, and the preceding PATH adjustments, it uses the python.org python3 interpreter.