I have a OSX Lion Server running as a file server in our network. But I have no clue if there's a way to use it's mailserver without smtp auth.
Sure. Turn it on.
The default mode is to not require authentication. You have to turn on authentication if you want it.
Changing authentication on/off is a matter of clicking a radio button.