Googlebot for smartphones crawler will follow robots.txt
Search engine giant Google is going to bring big changes on smartphone Use-Agent for content crawling on smartphones. Due to massive uses of smartphones or portable devices major web design switches to responsive style. So everybody has to think again different to make all online platform responsive style.
Those who are using Google blogger platform they are well known about Google webmaster tools. And earlier Google added an Smartphone option on webmaster tool. But the main work for crawling content Google use the User-Agent, however Google recently change this for crawling content on smartphone. The noticeable changes are-
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
But the old Use-Agent for smartphone is like below which is change soon.
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
For crawling your content perfectly you must update your Webmaster tools. Nevertheless don’t worry for your old User-Agent setting because new changes on user-agent string able to Fetch as Googlebot in Google Webmaster Tools.
Google also said about smartphones crawler that-
"The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile,”
Some webmaster unintentionally blocked their content for smartphone crawler with a disallow directive in the robots.txt on googlebot-mobile. But they have to unblock for smooth content crawl. Though this crawl will make small impact on our website but still we can increase our content visibility on smartphone search engine.
Anyway we can understand Google want to shape blog and website for smartphone search engine.
Now we require more improvement of all search engine for smartphone friendly, Google always in forth in this fragment. Every year searcher changing their search habit, as a result their search habit changes significantly from the last year. Most of them are using smartphone for search. You can see the below chart that is showing how smartphone search increasing.
Google Blogger already noticed that under Google Webmaster tools a new tab button option appeared where you can easily view the statistic or information about your web data as well as data about smartphone. Beside of this you can see the featured phone options that means which phones are using to view your blog or website.
Though Google is sloping higher on specific insights on webmaster tools and soon disclosing statistics about mobile specific URL errors. And it is possible to get precise SERP metrics for mobile search queries.