I tested this in Mavericks 10.9.4 but I suspect the big difference is what applications you have installed that have registered themselves as capable of opening these files. I as an example have an applications that can open ePub files installed (Pages) but not Mobi files.
With regards to using Unix, the standard ls (list) command would list them in name order not kind order. The ls command does have support for listing in date (i.e. time) order, or size order, but not as far as I can see file extension or kind order.
ls = name order
ls -t = last modified order
ls -U = creation date order
ls -u = last accessed order
ls -S = size order
Ok, as ls by itself will not do the job on Mac (apparently the linux version has an extra option that would which is ls -X) we need to use multiple Unix commands added together to do this.
ls | rev | sort | rev
This 'lists' the file names, pipes it to rev which reverse the order of the characters which means the file extension is now at the beginning, sorts it, then reverse the order again. It may not be perfect as epub gets reversed to bupe and this may result in unexpected sorting, but at least each file extension is grouped together.
Here is my list result, first the default name order
file.epub
file.mobi
file2.epub
file2.mobi
untitled folder
now sorted by 'extension'
file2.epub
file.epub
file2.mobi
file3.mobi
file.mobi
untitled folder
Clearly this approach could be extended further to perhaps improve it more.