Q: Using FCE 4.0.1

Before you go to the hassle and all the potential problems of upgrading the OS or setting up another laptop, first determine why it's"slow during transitions."

What is the media format?  FCE only works natively with QuickTime DV and AIC (and at specific resolutions and frame rates).  If you have any other format, you'll need to convert it prior to importing into an FCE project.

Does the media match your Sequence settings in FCP?  If no, then you need to convert the media to the proper format prior to importing - that alone will greatly speed things up.

Is the Sequence rendered?  If no, render the Sequence fully and see if the transitions are still "slow."

-DH

Very often the culprit with a system that has gotten sluggish is that your hard drive is too full.  Check the capacity & used space on your hard drive; if it's more than about 70% full, I suggest trimming things down to get your hard drive under 50% full.

Second, bumping your MBP to 8GB RAM wouldn't hurt, especially for rendering and realtime effects.

To the point of your specific question, in my experience FCE 4.0.1 works best with either 10.5.8 (which you already have) or 10.6.8.   I do not recommend moving to 10.7 or later versions of OS X if you want to continue using Final Cut Express.   FCE was discontinued before Lion (10.7) was released and many users have had difficulties installing and/or running FCE on Lion and later releases of OS X.

For what it's worth, imho, 10.6.8 (Snow Leopard) was/is the absolute high point of OS X releases.  I am still running it on most of my Macs.   I am ok with Lion (10.7.5) in general but not with apps that pre-date Lion, and specifically not with FCE; somewhat ok with Mavericks (although I do think it runs slower than Lion or Snow Leopard); and I detest Yosemite for quite a number of reasons.

I have plenty of HD space (130GB) left, so that's not the problem.

Actually, the amount of free space is not nearly as relevant to the issue as the % of free space.  If your HD is over about 50% full, especially doing video, there will be performance degradation compared to an HD that is less than about 50% full.  It's the physics of the hard drive.  In addition, if you are working on HD video you can easily need 50-100GB per hour of video for working storage & render files.  And if you render multiple times, FCE is not good at cleaning up old render files, so multiple renders take more & more disk space. The only effective way to clean out old render files is to manually delete them from your FCE  /Render Files folder.  And it's nearly impossible to tell which render files are actually in current use, so you end up having to delete them all and then re-render your entire timeline if you really want to free up disk space.

To answer your question about upgrading, yes, once you install Snow Leopard you should be able to update to 10.6.8 via Software Update.  That's how I've always done it.

If your black Macbook is the one I suspect it is, the official max is 4GB RAM but it appears it will work with 6GB.  Overall, the system specs are on the low side for FCE 4

As for still images, I have generally found sizing them to no more than 2x your video frame size works pretty well.  Larger than that, FCE will be discarding lots of pixels to fit the image into your video frame.   You need to consider the actual pixel dimensions of your image, not the embedded resolution or dpi.  Actual pixel dimensions are what's important.  The larger your jpeg image the more pixels will be discarded, so images that are much larger than your frame size are not advisable.