Q: Downloads folder in dock will not clear

I have a MacBook Air OS X Yosemite 10.10.1. The "downloads" folder in my dock has files that won't delete! I downloaded about 20 photos and when I went in and tried to clear the downloads folder only half of the files there got cleared out. Whenever I try to drag a file to the trash, it won't clear out. And when I click on "open in finder" it's empty and there's nothing there. So my question is how do I clear it out? It's just annoying when I download something new and I have this huge list that I don't need.