How to clear the cache memory?

Open activity monitor via spotlight click on memory , in the bottom side cached files is visible .

Cached Files: Memory that was recently used by apps and is now available for use by other apps. For example, if you've been using Mail and then quit Mail, the RAM that Mail was using becomes part of the memory used by cached files, which then becomes available to other apps. If you open Mail again before its cached-files memory is used (overwritten) by another app, Mail opens more quickly because that memory is quickly converted back to app memory without having to load its contents from your startup drive.

Cache memory aka swap files will be cleared when you reboot.

If that doesn't help, please post an Etrecheck report to look for other causes: http://etrecheck.com