But If I keep one connected and then just pull it and put the other one on once a week it wouldnt go back all week then but rather just 20m to when I pulled the other?
Not sure if I understand what you are asking.
If you disconnect the "regular" drive that you are using for Time Machine backups, and connect the drive that you want to use "weekly", then Time Machine will back up to the "weekly" drive that you have just connected.
The first time that you connect the "weekly" drive, Time Machine will back up all of the data on the Mac. After that, Time Machine will back up the changes that have occurred on your Mac since the last backup.
If you don't want to wait for a hour or so before the backup will actually occur, you can click the Time Machine "clock" icon at the top of the Mac's screen and click on Back Up Now.
The backups to the "weekly" drive are going to take longer than normal backups to your "regular" drive since Time Machine will back up all of the changes that have occurred on your Mac since the last "weekly" backup has run.
Once the backup to the drive that you will use "weekly"has completed, and you disconnect that drive and then reconnect the "regular" drive, then backups will resume normally again within an hour. If you don't want to wait for up to an hour for the next backup to run on the "regular" drive, you can use the Back Up Now option that I mentioned previously.