Remove fill collor

Good day.

In Numbers there is the option to pre-set fill colors so that cells can be filled with a pre-set color easily, see the picture.

In the past the box with pre-set colors has been filled with some not-used colors. I want to remove those not-used colors but I can't find the option to do so.

So the question is: How can I remove pre-set colors from the fill color box?

Thanks in advance for your answer and/or feedback and with kind regards,

Herman