Many webmasters have started to see the appearance of a rather disturbing warning message from Google in their emails recently.
The message essentially tells webmasters that Googlebot is having difficulty access the website as they cannot ‘access your JavaScript and/or CSS files because of restrictions in your robots.txt file.”
The warning message so far seems to be confined to owners of WordPress websites and from early forum chatter this seems to be something structurally about WordPress rather than an individual Webmaster issue.
The worrying element of this is that Google has specifically written that this “affects how well our algorithms render and index your content”
Since when has it been a requirement of any Search Engine to have to access your Java Script and CSS files to figure out what a page is about? The simple answer is that they haven’t needed it and don’t need it and this development, if unchallenged, gives Google access to what could be proprietary or copyrighted information and more importantly lays that open for all to see.
As this development is fairly new we suggest that you don’t do as Google ask and make the changes they require, despite them implying that it may damage your search rankings.
Firstly, if Google are going to harm a websites search ranking through not complying with their latest ‘request’ then this becomes a much bigger issue. They do not need access to your CSS or JS files to rank your pages and their latest request is intrusive.
Secondly, it seems to be specifically a WordPress issue which means you are not alone; this affects almost a quarter of the worlds websites.
Our advice therefore is to hold hard and do nothing. Sure the temptation is there to try and ‘fix’ your website but this is one occasion where either Google will back down or, if a fix really is needed, then it will come at the structural level and you can expect the update from WordPress fairly shortly.
Remember that amongst other things, your robots.txt file is there to allow you to specifically block certain elements of your website from the Search Engines and if they now want you to open the doors and let them have access to everything then frankly it may well be time for all of us to give up and go home.
