I'm not sure, since I haven't tried RVM yet. Looks interesting though. I've dabbled with it, but not in depth.
Safety all depends on what you know about Linux, and I'm sure you know quite a bit, since you are programming Ruby.
I just tuned up a few things on my system lately and realize that out of the box OSes and Apps, and the internet in general, is not secure. The enduser has to figure that out for him/herself
(ex: suid, sgid like rlogin rsh are set with the following permissions and the Disk Utility's "Repair Permissions" suggestions are generally quite good. Not so good on the rlogin, rsh ... programs though.
I guess you could say the same applies to your compiling your own program (ruby src). If you understand how ruby works, you can set it up, choose the options that you need and have a "safe" ruby.
I do however like how port sets things up. It's generally quite clean since it finds and knows what which dependencies to install.
I'm sure you understand this.