You cannot remove any items from smart albums, without deleting them from the library. You have to change the rules of the smart album, if you want to modify it.
Smart albums are automatically populated by photos, based on all photos in the library and the rules of the album.
Why do you want to empty the album? Are you trying to save storage? Albums do not store photos; all photos are stored in Photos only once. Albums are just referencing the ne copy of the photo in the library. Adding photos to albums does not move them or create duplicates. You can add the same photo to many albums without needing extra storage.