Google has this way to verify that a bot accessing the server really is Googlebot by using a reverse DNS lookup. (https://support.google.com/webmasters/answer/80553?hl=en)
Is there some way to do that for AppleBot?
Right now if I do the same for AppleBot I don't get anything.
Maybe things have changed since last year. 🙂 Testing today, real Applebot does reverse resolve in the way described in the Google article. Here is an example from today (some data removed):
[user@localhost logs]# host 17.###.###.###
###.###.###.17.in-addr.arpa domain name pointer 17-###-###-###.applebot.apple.com.
[user@localhost logs]# host 17-###-###-###.applebot.apple.com
17-###-###-###.applebot.apple.com has address 17.###.###.###
And an example web server log entry:
17.###.###.### - - [16/Feb/2016:00:00:00 +0000] "GET / HTTP/1.1" <return code> <bytes> "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)"
I left 17. in my example as this is also a characteristic of Applebot, per Apple.
Is there some way to verify the AppleBot, maybe by a reverse DNS lookup?
