Opening a picture that is smaller than the window will size it up to fit just like opening a picture that is larger than the window will size it down to fit. This is the expected behavior in Photos and I do not know of any way to override this.
You could re-size your Photos window to a size closer to that of the small picture that you're opening.
You can right-click on a photo and use the Edit With command to use a different app to open it such as Preview which would not automatically enlarge it to fill the screen.
I realize that neither of these are great options but that's all that comes to mind.