How do I customize the order of emojis and symbols in macOS?

I often use the inline "emoji and symbols" feature (by pressing ctrl+command+space, for example). Since sometime around 2024, a group of emojis always comes to the top, which bothers me to reach "favorites" and "recent items."

As far as I can see, there are no "recent items" that I actually use are shown. Instead, those emojis appear in the "recent items" section even though I did not use any of those emojis. "Favorites" comes after those emojis.

The actual recent items are shown on the bigger screen (although it is in Japanese):

I already tried to delete emojis on the bigger window of "emoji and symbols," but that did not work.

Is there any other way to change the arrangement of the items to show the actual "recent items" or "favorites" on the top of the list?

Thanks,

[Re-Titled By Moderator]