Tom,
A couple of things ...
First is -- as you correctly point out -- you cannot legally use an rtmp:// URL scheme within an RSS enclosure:
http://www.rssboard.org/rss-specification#hrelementsOfLtitemgt
Though I cannot say with absolute certainty, I am willing to buy Duncan lunch if Apple isn't validating RSS feed XML. So it is extremely unlikely you're going to slip one past Apple. 🙂 🙂
Next thing is, lessay you've got a legitimate RSS server doling out Flash files (that is, a server that feeds in the normal way). Further, lessay you craft your enclosure to specify the correct MIME type for Flash:
application/x-shockwave-flash
You'd then have an enclosure that looked something like this:
<enclosure url="http://www.myflashserver.edu/flash-videos/MyMovie.swf" length="12216320" type="application/x-shockwave-flash" />
Okay, so the problem now is that you won't be able to get iTunes to respect the enclosure because iTunes has no Flash plugin ... Flash is not a supported media type in iTunes. 🙂