Time to upgrade my Mac Mini?

I totally agree with Rudegar your first step should be to update the amount of RAM.

You could upgrade to 8GB which would probably be enough to make a noticeable difference but you might as well also look at the cost of upgrading to 16GB. If the difference is small you might as well go to that level.

Once the RAM has been upgraded the next option to consider is if your Mac mini has a hard disk rather than an SSD drive you could replace the internal hard drive with a 2.5" SATA SSD drive.

You are fortunate to have a 2012 model rather than the 2014 model as the RAM cannot be upgraded in the 2014 model.

Memory and then a ssd drive , you would be amazed at the difference it will make.

Speed is a relative thing . before I upgraded this 2104 iMac I though it was pretty fast . The I bought a 2019 Mac mini with 64 gig ram and a 1 Tb ssd drive and was amazed and the speed difference . I upgraded the iMac to 16 gig (max it would hold ) and A ssd drive and its now almost as fast as the Mac mini (almost).

You can confirm how much RAM your computer thinks you have by going to the Apple menu and selecting 'About this Macintosh'.

If you want to actually test the memory chips then whilst Apple have a built-in hardware diagnostic tool it is both very limited and often missing due to the way some people may have installed or upgraded their Mac previously.

The best free option is to use Memtest86+ which can be placed on a bootable USB drive. (It is not a Mac program but rather a Linux one.)

See - https://www.memtest.org/

or - https://www.memtest86.com/download.htm

This article helps explain how to create the USB boot drive - https://www.idownloadblog.com/2017/06/08/test-ram-integrity-memtest86/

4gb is the cause you should be able to upgrade it

https://www.crucial.com/usa/en/compatible-upgrade-for/Apple/mac-mini-2*5ghz-dual-core-intel-core-i5---late-2012