It sure sounds like you are performing the steps correctly so try quitting the Newsstand app completely, reboot the iPad and then try the steps again to delete - but perhaps rebooting will cure the issues without having to delete again.
Go to the home screen first by tapping the home button. Double tap the home button and the task bar will appear with all of your recent/open apps displayed at the bottom. Tap and hold down on any app icon until it begins to wiggle. Tap the minus sign in the upper left corner of the app that you want to close. Tap the home button or anywhere above the task bar.
Reboot the iPad by holding down on the sleep and home buttons at the same time for about 10-15 seconds until the Apple Logo appears - ignore the red slider - let go of the buttons.