How do you troubleshoot EarPods not charging when in their case?

My AirPod case charges just fine, but intermittently a single EarPod will not charge when in the case. This results in having one EarPod fully charged and the other with no power. If I open the case and wiggle the EarPod, the light indicator will change from green to orange to show it's charging, but if I remove my hand and set the case down the EarPod seems to loose connection and stops charging.

Anyone else experience this?