Some DNLA software aka. UPnP software includes the ability to 'transcode' video and/or audio files from one format to another. So if you have a mixture of DNLA clients which do not all support the same format it can convert the files for those that need it. You may want to take this in to consideration when choosing a DNLA package for your Mac.
Before I get on to DNLA I would say that DNLA is old, poorly supported, poorly featured and I would advise if possible to avoid it. There are other approaches which you may decide are better for you but if not then move on to the DNLA answers below.
Firstly, other non-DNLA approaches. The premier approach for doing multi-room audio is to buy a Sonos system. With this you either buy Sonos speakers e.g. Play:3 or you buy a Sonos Connect which allows you to plug in to an existing amplifier. Sonos supports all the audio formats supported by iTunes including Apple Lossless which is going to be the best format to use if possible. While Sonos is the market leader there are other brands offering similar solutions. Examples of other similar systems to Sonos include Logitech Squeezebox, Roku SoundBridge (discontinued), an Apple TV3 or TV4, etc. All these support Apple Lossless and all allow different rooms to play different tracks at the same time.
A second approach is to use Apple's AirPlay standard, many AV Receivers now support AirPlay if you have a network interface on your AV Receiver which these days is either standard or an option on most. iTunes would then stream directly to the AV Receiver using the AirPlay standard.
A third similar approach is to get one or more AirPort Express WiFi base-stations, these have an audio out connector, you AirPlay from iTunes to the AirPort Express and it outputs via its audio connector to the attached audio device which could be speakers or an AV Receiver.
Now there is a limitation with AirPlay, a single iTunes can only output via AirPlay the same content to one or more AirPlay destinations, if you want to play multiple different music tracks at the same time then this is where Sonos and similar solutions win hands down. The iTunes however can be a Mac, or Windows, or an iPhone, or an iPad.
Moving on to DNLA. You are probably going to find many DNLA servers for the Mac are either discontinued, or semi-discontinued in that they have not had a recent update and therefore may have problems with newer versions of OS X, or only work with some clients, or do not do Transcoding well or at all. In other words like DNLA as a whole it is rather a mess. You will therefore find that you may have to buy a commercial DNLA package although this will be still cheaper than say buying a multi-room Sonos setup. I would advise testing the trial versions, make sure to remove each before trying the next.
With no particular sentiments, have a look at the following.
http://www.cyaneous.com/products/firestream/
http://www.universalmediaserver.com/
http://www.isedora.com/
http://twonky.com/
http://serviio.org/
http://www.tvmobili.com/
http://www.tonido.com/app_mediaserver_home.html
