"This has layers in them [something which Photoshop also has and I am afraid of!] "
Why are you afraid of applications that use layers? Layers are one of the easier concepts of both drawing and painting apps to understand and use. Think of layers as multiple sheet of clear paper or thin sheets of clear plastic that you can draw or paint on. You can build up a drawing or painting by putting different elements on different layers. If you want to alter elements on a layer only that layer gets affected. If you don't alike what you did on a layer, you can erase an entire layer and start drawing or painting on that layer, again, or delete the layer entirely. You can, also, more easily move elements around on a layer more easily that having all the elements on a drawing or painting on one layer.
So, there's the quick tutorial about layers.
What type of diagrams are you drawing/painting?
There are, also, diagramming programs, that have make drawing simple geometric shapes easier to do and easier to move around and esier to edit and change their appaerance. If there is text within the shapes, both the shape and text moves as one unit, saving you from moving the shapes and text as separate elements.
Here are two diagramming apps for example
Shapes
http://shapesapp.com/
OmniGraffle
http://www.omnigroup.com/products/omnigraffle/