It's not. OS X manages the memory by taking control of most of it so that it is immediately available as needed. This is normal. Don't sweat it until you are running out of memory because you really are running too much concurrently.
About OS X Memory Management and Usage
Using Activity Monitor to read System Memory & determine how much RAM is used
OS X Mavericks- About Activity Monitor
Understanding top output in the Terminal
The amount of available RAM for applications is the sum of Free RAM and Inactive RAM. This will change as applications are opened and closed or change from active to inactive status. The Swap figure represents an estimate of the total amount of swap space required for VM if used, but does not necessarily indicate the actual size of the existing swap file. If you are really in need of more RAM that would be indicated by how frequently the system uses VM. If you open the Terminal and run the top command at the prompt you will find information reported on Pageins () and Pageouts (). Pageouts () is the important figure. If the value in the parentheses is 0 (zero) then OS X is not making instantaneous use of VM which means you have adequate physical RAM for the system with the applications you have loaded. If the figure in parentheses is running positive and your hard drive is constantly being used (thrashing) then you need more physical RAM.
Adding RAM only makes it possible to run more programs concurrently. It doesn't speed up the computer nor make games run faster. What it can do is prevent the system from having to use disk-based VM when it runs out of RAM because you are trying to run too many applications concurrently or using applications that are extremely RAM dependent. It will improve the performance of applications that run mostly in RAM or when loading programs.
If you open Activity Monitor, click on the Memory header to display usage in descending order, then look below at Memory Pressure. When it's in the red most of the time then you need to add more RAM.