Clearing Swatches from Apple Color Picker

I use a few programs where I make extensive use of the Apple Color Picker, often using the Shades module in preference to the stock varieties (crayon, color wheel etc.) At the bottom of any Apple Color Picker window is a drawer which allows drag and drop color swatches. It's very handy to use but often, after finishing an image or project, I wish to simply clear all those swatches. I've never figured out how to do that. All I can figure is to drag a white 'swatch' on top of a colored one to 'empty' the swatch.

Does anyone know a keyboard command (no contextual menu is available) to simply clear all the swatches in a color picker window with a single command?

Thanks for any thoughts on this rather obscure point.