apple id compromised- not my phone number

I don't remember how this started but I got a message that read "Is this still your phone number? Current trusted number:" I didn't click on the button to change the number. I went to my settings and saw that the phone number was indeed not my number.I texted my brother to ask him what he knew about this. The next day, when I looked at my settings, the phone number had mysteriously changed back to the correct number. I have had serious concerns of someone harassing me. I believe that is what has happened here. I am wondering what I should do. I guess I can change the password but things like this just keep happening. The password was not something anyone could guess. The person harassing me has plenty of money, access to very tech savvy people and a lot to loose himself.

Do I just have to live with this? I have nothing to hide so I guess I shouldn't worry but it is unnerving.