Try resetting the iPhone. Hold the on/off and home buttons together for about 10 seconds until the Apple logo appears.
The keyboard must be appearing because you have some app in the background awaiting input. Hopefully the reset will end that situation.