I assume you are talking about in an album - since you cannot change the order in the all photos view at all.
In an album, you can sort by date (ascending or descending) or by title. You can drag and drop into a specific order, but as you have found, it is not very sticky. If anything causes the album to resort, then it will go back to whatever is defined as the sort order for that album.
The best (and only real) solution if you want a specific order is to give the images titles with a numeric prefix. The titles are sorted alphabetically, not numerically, so in order for the title sort to work correctly you have to have enough leading 0's for all the images in the album to have an alphabetic order - so if you are never going to have more than 1000 images in the album, you can get away with 3 digits. EG:
(Note, in all the examples, you could have anything you wanted after the prefix - eg 00001 Jenny, 00002 Rome etc)
000
001
002
003
...
997
998
999
However, if you then went over 1000, it would then fail to sort properly and you'd need to manually add another leading 0 to all images. So if I were doing this, I'd add an extra couple of leading 0's above what I thought I needed. EG:
00000
00001
00002
...
00997
00998
00999
Then if you unexpectedly needed to go up to a larger number you'd have no problem.
01000
You could even choose to go up in steps of 10 rather than 1 which would allow you to insert images into the sequence between two other images later - which would require a complete renumber of all later images without the space included from the start:
00000
00010
00020
00030
...
09970
09980
09990
But if you do this, you need to consider whether you need another leading 0.