There are several reasons.
First, the storage controllers themselves. The interface for the SSD does a lot more than the SD card and therefore takes up more space. The controller in the SD card basically only translates the data stream, puts what it can in a tiny cache, then writes it to the directed sector. That's it. The SSD does that plus manages bad blocks, performs wear leveling, does garbage collection (erases formerly used blocks that are now unused when idle and maps them back into usable space) and other maintenance and monitoring functions.
Second, to achieve the ultra high speeds, SSDs use a somewhat different physical structure with their storage than SD cards to achieve these much higher speeds which takes up a bit more space.
Third, also related to speed, is the cache size difference between the two. Since writing directly to the storage cells is somewhat slow, higher speed cache memory is used which can buffer the data before writing to the actual storage. In the case of SD cards, that is usually quite small, typically being at most a few hundred kilobytes to maybe a couple megabytes. On the SSD, the cache is on the order of gigabytes due to the high speed demands od operating systems and computer software.
Forth, also related to speed, the size of the SSD needs be bigger in order to dissipate the heat that is generated. Given that the fastest SD cards are around 300 MB/s, the NVMe SSDs are 5 to 20 times faster and with speed comes heat.