You should connect your backup drive more often that once a week.
Time machine uses a system data structure called the File System Event Store to quickly create a list of changed folders. This shaves about four hours off the elapsed time of all but the very first backup. If the File System Event Store is stale (which it often will be after a few days without a backup), then Time machine must do a "deep traversal" -- look at every existing file backup date and compute what needs to be backed up from scratch.
Time machine is optimized to run at low priority in the background, without disrupting your important work. If you just let it run, it will finish a routine incremental backup in only a few minutes and quit. If it has to do a deep traversal instead, it will spend hours determining what has changed before copying any files.