I actually figured this problem out. I'm running a server that's running Fedora Core 5 and I couldn't figure out how to use it headless with ARD. I had been using a VNC client. In the VNC client, I was connecting using virtual display 1 (display :1) but couldn't figure out how to do this in ARD as ARD defaults to use display 😮 - that's how the end user sees their mouse and keyboard movement - display 😮 is the actual display connected to the computer.
Through some poking around the web, I found this wiki article
http://en.wikipedia.org/wiki/VNC
(look at the How it works section)
It explained that the virtual display ports directly correspond to the VNC port numbers. Display 0 is port 5900, display 1 is port 5901 etc on up the line. So, what I did was after adding my server to my computer list in ARD I then edited the computer and changed the IP address so that I added :5901 on the end, designating the port number. So, my IP address of the server now reads as 192.168.100.5:5901 in ARD.
WORKS LIKE A CHARM! Granted, you only get standard VNC usability and not the nice things that ARD provides, but it's way better than having to fire up a separate client for admining 1 box.
Hope this helps someone!
PowerBook G4 15/1.67Ghz/512MB/80GB/128MB Video Mac OS X (10.4.1) 
