So was what you downloaded an app, or just a link in Safari? Did you provide any personal information along the way, such as credit card, social security, name, address, email address, Apple account information?
If it wasn't an App and you didn't give whatever it was you accessed ANY personal information, it's very unlikely they were able to access anything on your phone.
I do agree with MrHoffman and suggest you change passwords for any applications which have important access for you.
Otherwise, iPhone iOS is a sandbox operating system which is very difficult to penetrate, unless your phone was jailbroken, but you already confirm was NOT.
Definitely go to Settings > Safari > Clear History and Website Data at a minimum. I do hope this reminds you to be careful with any computing device and not click on sites which you are not familiar with, regardless of whether it's on iPhone or a computer. Overall, I suspect you're just fine.