For me the main difference is sheer simplicity. With the appleTV it takes a couple of minutes to setup and then it's good to go. At some point with a mac mini, you're going to have to hook it up to a keyboard and mouse and actually configure the thing. As someone else mentioned, that's just one more box to maintain.
I'm assuming you already have some other mac which you intend to use as your media server (otherwise your appleTV won't be much fun!). You can set up your MacMini with some aliases pointing at the movies and music folders on your media server and then you dont have to worry about adding content to the MacMini.
The MacMini takes a lot more work, but once configured it can deliver many more video formats than the appleTV and also has the HUGE advantage that you can stick a DVD or a CD straight into it and play it, without needing to rip-it. Newer ones can also deliver 5.1 sound.
Where AppleTV has the upper hand is that it won't require maintenance and the interface is also much faster than FrontRow.
Basically depends how much time you're prepared to spend "tinkering" with things I guess...
MacPro, MacBook Mac OS X (10.4.8)