mathtag.com is the domain of a company called Media Math, they are an internet advertising and marketing company, so it's not uncommon for their content to appear on other sites. Normally this all happens without error, although in this case my guess would be that mediamath had not set up the security certificate correctly on the site, so an error message occured.
This kind of thing is not uncommon and usually occurs because of a mistake made by the site's owners, it's nothing to do with your own machine. It pays to click no in these cases (generally the connection is to an image or other content linked on the site) just to be safe. Mediamath are legit so no real chance of any of your personal data being compromised in this case.