HT204683: About Applebot

Learn about About Applebot
daniel_morrow
Level 1 (4 points)

Q: how will enabling "robots.txt" affect "non-production" or testing servers?

So, say my website is:

 

"https://www.myspecialwebsite.com". This is the website that my users connect to. And this is the website that has all of the data that my published app uses.

 

I also have another website:

"https://qa.special1.com" - which is only used for QA purposes. While it's not behind a firewall, it's not well known, either. (note the change in domains, btw).

 

In my Xcode project, I have two targets for the same app. The first one is the "PROD" target, that points to "www.myspecialwebsite.com", and the second one is a QA target, that points to "qa.special1.com". The second one is only ever used by people in my company, testing the app. When it gets closer to release, we switch to using the PROD build.

 

The point is, if I allow AppleBot to crawl my "qa.special1.com" website, will that cause problems for me?  I'm implementing Core Spotlight, web markup, NSUserActivity, etc. I guess I'm just wondering if this stuff can be tested on the "QA" server, without affecting search results in the PROD server?

Posted on Jun 29, 2016 1:01 PM

Close
daniel_morrow

Q: how will enabling "robots.txt" affect "non-production" or testing servers?

  • All replies
  • Helpful answers

  • by etresoft,

    etresoft etresoft Jun 29, 2016 9:20 PM in response to daniel_morrow
    Level 7 (29,380 points)
    Jun 29, 2016 9:20 PM in response to daniel_morrow

    Hello Daniel,

    What do Core Spotlight, web markup, and NSUserActivity have to do with your web site? Any web crawler should only access public data anyway. If you are posting public data to a server and allowing robot access, then it may affect search results. I suggest requiring an obscure query parameter for any access to your QA site. That way, anyone who isn't "in the know" will probably not see anything but internal users and apps can still access it. There shouldn't be any security issues because the data is public anyway. Or, you just go right ahead and require a user account and password to access the QA site at all.