I was having a similar issue and found your post while trying to resolve it. Everytime I right-clicked on the iTunes Store link, the "Open in a New Window" option would flash on the screen but not long enough to click on it. Then it would just open the iTunes Store on top of the library. After looking at a few other posts for some hints, I searched my iTunes menus for options.
Under the View menu, I unchecked "Use Full Window for iTunes Store." Now when I right-click on the iTunes Store link, it still opens it up automatically in the current window, however, because it's not using the full window anymore, I can right-click on the link again and it will let me select the "Open in a New Window" option. Then I can open it up in a separate window from my library. A bit of a round-about way to do it but it's closer than I was before. Hope this helps, if you haven't already found a better way!
I did this on a Windows XP SP2 machine and am using iTunes 10.2.2.12 (current version for my setup).