It is not clear to me what you are doing, but I am suspecting that you may not be defining it correctly.
" I carefully put in all the email addresses I wanted it to load putting as you suggested (any recipient) but loaded a whole host of stuff irrelevant so I'm thinking do it manually."
For one thing, I never suggested to use "any recipient".
It makes sense if the criterium you want to check against for example the recipient has a particular address.
Note that if using "Any", a message needs only match one criterium to get in.
That may be why you are getting a lot of messages in the smart mailbox.
Regarding flagging the message: you can create a smart mailbox based on flag:

but AFAICT you can't select the messages with a flag of a particular color.