Do you know the subnet the camera is in? Knowing this will help but you can still get around without it.
If you know the subnet (e.g. 192.168.0.x/255.255.255.0) then:
1) connect the camera directly to your Mac's ethernet port.
2) Use System Preferences -> Network to configure your ethernet port with an address in the same network (e.g. 192.168.0.1)
3) Open /Applications/Utilities/Terminal.app and run: ping 192.168.0.255
The address you ping is the broadcast address of the network, so adjust if you're not using 192.168.0.x.
What should happen is you get a reply from each device on the network. Since you only have two devices (the Mac and the camera) it should be pretty easy to work out the camera's address.
If you don't know the subnet, you can also try pinging 255.255.255.255
This is the universal broadcast address, but not everything will respond to it.
Finally, if that doesn't work you can try a
tcpdump to identify traffic on the network, but it'll take a little decoding. If you configure your ethernet port as 192.168.0.1 then the following command in Terminal.app will show any traffic from any other device on the network:
sudo tcpump -i en0 not host 192.168.0.1
You should be able to see broadcast traffic such as ARP from the camera, from which you can determine its IP address.