Both the mini-displayport plugs and TB plugs are physically compatible which is why you can plug it in. But if the iMac has a mini-displayport for video it won't work with the TB display.
Check your System Profile to see if it tells you whether you have a TB port. And if there is no information about that post here or look up for yourself the kind of ports for that model of iMac. System Profiler will give yo that information (click Hardware on the left). From the model information it can be definitively determined whether that imac supports TB.
Also, so long as the Displays system preferences is not showing an Arrangement tab then it is not detecting the second display.
The Thunderbolt connector is indeed the same physical shape/size as the older Mini Displayport. Any Thunderbolt-enabled Mac can run both the Thunderbolt display and any of the older Mini-DP displays; unfortunately, only a TB-enabled Mac can run the Thunderbolt display, even though its connector will appear to fit just fine.
Here's an article comparing the Thunderbolt connection (lightning bolt icon) with the older Displayport (square with two vertical lines either side):