Not sure what shape you're trying to make but the custom shape tool ( I presume you mean the Draw tool, which looks like a pen nib) can make just about any shape. It works in similar manner to like tools in Illustrator etc.
Are you click-dragging when you get to a vertex or point (to introduce bezier handles). This allows you change the curves, or remove them.
Also, select the shape and go to Format -> Shape -> Smooth Path may help, and Format -> Shape -> Make Editable. Holding down the Option key and clicking on the side of a shape allows you to add more points to a shape.