My phone was exhibiting this as well, but it also lost a screw on the bottom. I took it to the Apple store and the replaced it. Either issue would have warranted a replacement, I was told. Now my second phone is also exhibiting this issue with the camera.
As someone pointed out, this is not something that is easy to see affecting the camera, but it will. It causes a general softening of the entire image. The amount of softening is proportional to the fraction of the opening that is obscured. That whole area is needed by the camera. They don't cut out extra space that's not needed. So if you can see any reasonable amount of obstruction, it's affecting the image. It may be hard to notice, but your images will have lower contrast and softer edges, and it will gradually worsen.
And regardless, people DO buy Apple products because of aesthetics, and Apple certainly takes pride in the aesthetics of their devices. If you take a product to Apple which is aesthetically flawed because of manufacturing, (not usual wear and tear over time) they are required to help not only because of the wording of their warranty, but by their own standards of business. Since this manifests within days/weeks of getting a phone and is entirely internal, it's pretty clear it's a manufacturing defect and everyone who is bothered by it should be getting it fixed.
If it doesn't bother you, then great, you don't have to go through the hassle of restoring your phone and swapping it out. But don't go telling others they aren't entitled to get what they paid for. I could care less if it's a refurb or not, as long as it's fully functional and aesthetically as advertised.
I'm going to give mine a little longer to see how much worse it gets, but I'll be heading back to the Apple store about it.