Private Browsing Mode

I seem to be unable to get out of private browsing mode. In safari, I touch the icon down at the bottom right corner (looks like two boxes overlapping one another). The Private button is at the bottom left. It has a box around it, private browsing mode is in use. When I touch the private button the box goes away, but I am still in the same mode. Please help this great-grandmother is lost......