Yes, it will work. But, you cannot use it with a SIM card from the Japanese carrier (Softbank), since all iPhones sold in the US are carrier-locked to AT&T. That means you'll be on international roaming, and likely run up hundreds or thousands of dollars in roaming charges (I'm not kidding).
Honestly, you're better off buying a pay-as-you-go phone in Japan and using that for your calls. You can get them from vending machines on street corners. If you want to bring your iPhone, set it to Airplane Mode and leave it there, you can turn Wi-Fi back on and use that (consider Skype over Wi-Fi, for example).
You can also get a Google Voice number so calls to your US number can be forwarded to your Japanese phone.
If you really want to take and use your iPhone as a phone, see these:
http://wireless.att.com/learn/international
http://www.wireless.att.com/learn/international/roaming/iphone-travel-tips.jsp