This has nothing to do with background app refresh. Even with background app refresh turned off for an app, that app will still remain in memory and paused when you switch to another app. If iOS determines that memory used by a paused app is needed by another app, it will signal the app to save its state and terminate.
A properly written app will do exactly that. It will save its state to permanent storage and terminate itself. The next time the app is started, it will restore that state and pick up exactly where it left off. But only if the app was written to do so. Many apps, particularly games, are not written to do this.