The lossless compression is not the issue. Note on page 87 of the Users Manual for the D800 (love mine) that the print size at 300 dpi is 24.5 x 16.4. So with larger prints than that, the resolution must drop below 300 dpi, which depending upon the detail of the scene photographed may be responsible for your "bluring". Btw, on that page the FX (36 x 24) is reported as 7360 x 4912 -- where did you arrive at the 8390 number? But the number of pixels is fixed by the camera's sensor.
The latest versions of Photoshop can use vector scaling to upsize photo files for large scale printing. I have not tested how (or whether) Aperture might perform at scaling an image to add pixels.