Hi,
In the past in the Mountain Lion (Messages 7.x.x) I have sent quite large photos via the iMessages account of several GBs of data.
I have yet to do the same in Messages 8 (Mavericks) but don't expect for this to have changed.
I do know that the Messages 8 app tends to zip some files before sending when dragged from other apps such as a Browser or iPhoto.
I have sent a Keynote file to my iPhone in Messages 7.x.x but this was only 334 kb
What sort of files are you trying to send ?
If not zipped already it may pay to try this first.
7:21 pm Sunday; November 3, 2013
iMac 2.5Ghz 5i 2011 (Mavericks 10.9)
G4/1GhzDual MDD (Leopard 10.5.8)
MacBookPro 2Gb (Snow Leopard 10.6.8)
Mac OS X (10.6.8),
Couple of iPhones and an iPad