Back in October 2011 under the guise of ‘making search more secure’ Google started to remove access to keyword data in Analytics and replace it instead with the dreaded (not provided).
From a position where we could all see what search term had triggered a customer landing on our website, we were slowly facing an ever darkening landscape where we had no ideas what people had typed in. Keyword data was being hidden from everyone in the name of privacy. At the time, it seemed like a slow creeping blindness which hampered our ability to optimise websites.
Today we can see barely 10% of the search queries in Analytics if we are lucky with the rest all hidden.
Of course, if you run a PPC campaign you still had the ability to see this data so why Google chose to exclude it from Analytics when it was still freely available on their paid platform has always been something of a mystery. It’s not as if more privacy was protected if you paid for the data?
The good news however, is that this data is now available again, but it is slightly hidden away.
Now granted, what we are about to show you isn’t an exact replica of the old keyword data but it is a very useful resource and without access to paid keyword data this is probably the best available free resource we know.
If you want to find it this is where you go….
Webmaster Tools is your new friend!
Recently we have found a few new customers who hadn’t bothered setting up Webmaster Tools, or if they had set it up then they hadn’t bothered using it at all.
This is a big mistake.
Google Webmaster Tools (WMT) is a fabulously rich resource for any Webmaster or website owner and if you don’t already have an account linked to your website then set one up now.
When your account is set up then log into Webmaster Tools and navigate to ‘Search Analytics’ on the left-hand navigation, which is the first link under the main heading of Search Traffic. Search Analytics first appeared in its current guise in May 2015 and Google has been tweaking it ever since making it better and better for Webmasters.
By default, you will see the search analytics dashboard defaulted to ‘clicks’ and ‘queries’, giving you the number of clicks you have received and from which search query, which, in essence, is the same as the old keyword data in Analytics.
The key difference here though is that whereas you previously got the exact keyword that people typed in now you are getting a much broader search query, some of which will be guided by Google’s autocomplete.
Now at this point, it is worth pointing out that the data is not exactly the same. It never has been between Analytics and Webmaster Tools and this recent change to WMT whilst welcome, does not totally replace the previously accurate Analytics data. What it does do is to give you a much better idea of what people are searching for when they come to your website.
The good news though is that it gets better from here.
Let’s say for example that you want to find out which of your landing pages are receiving the most traffic and which keywords are driving that traffic, as we all used to in the ‘good old days’. You can find the first part out from Analytics but you can’t see which keywords triggered the search.
Now in Webmaster Tools you can see that data.
If you click on ‘pages’ and select the date range you want on the right-hand side, it will show you a list of your top pages, the ones that received the most traffic during the time frame you have chosen. It is worth pointing out at this stage that we have had to anonymise the data in these next images at the request of our customer, but you can still see all the relevant elements.
If you then choose the page you want to investigate, click on the double arrows on the right-hand side of the ‘clicks’ column it will show you this;
Now at this stage when you see “Table omitted as it contains no additional information. Select Pages to see the top pages for the current results” you might be tempted to give up thinking there’s nothing more to see but….if you now click on ‘Queries’ you get this;
It has taken the data available for that page and broken open the keywords and phrases that have triggered the click in the first place. So in this example, you can see that the first keyword query has generated 96 clicks, the second 84 and so on.
You can also drill down further using the drop-down on the ‘Pages’ option to compare pages, so you can see if there are any common keywords between two pages or indeed any similarity in terms of patterns of usage,
Just fill in the URL you want to compare the first page against and this is what you get;
In this case, we found that the spikes in keyword 2 all happened on a Sunday which given that the post was about a recipe for Sunday dinner made perfect sense!
You can also expand the data available to you by clicking on the top tick boxes so you can see not only Clicks but also Impressions, CTR and Position.
In this example, we’ve ticked all of the boxes but the end result looks a bit confusing so our recommendation is to simply compare just two variables at a time as this is much easier to both read and interpret.
As usual with data from WMT there are some confusing bits, namely that this data will not correlate directly with Analytics data nor is there any way of independently verifying this data unless you want to spend hours in the server logs. The other point which often confuses people is the ‘position’ as this often bears no relation to where the page sits for a keyword query.
The reason for this is that what you are seeing here is an average keyword position over the timeframe you have chosen, meaning that if Google has shown you at position two for a search made at 10.00am it might show you at position 3 for one made at 10.05am simply to understand better how consumers interact with your and other websites. Ther is no guarantee that you will keep a constant position for any particular keyword search.
On top of this, there is always the case where people misspell and Google autocorrects and that may show you in a slightly different position to the one for a correct spelling. Overall however if you accept that it gives a broad view on where you rank relative to your competitors for this search query it gives you an idea on whether your content is good enough or needs improving.
With a decent data set, you can then start to make up your own metrics linking keyword position and clicks and CTR by client which is hugely helpful.
So the keyword data that we’ve all been craving since 2011 is now available again.
Now there are a couple of caveats with this which we encountered when using this feature in Webmaster Tools.
Firstly there is no guarantee that the data is accurate. For a start, the only people who verify Google data are Google, so there is no independent tool to say whether this is correct or not. Even if you choose to go through the server logs I guarantee that you will find differences so it’s a question of whether you want to spend the time on this. We recommend simply running with the Analytics data as even if it is wrong then at least you have a set of relative data to work with.
Secondly, when we clicked into the data for some smaller clients we found that it might say for example that the page had received 46 clicks yet the keyword data added up to just 6 clicks. There is no additional information available on the site to explain why there might be this discrepancy but when we tested we found that pretty much every customer had the same issue.
Finally, when we compared this data with some PPC data we found major differences and given that Google is obliged to show accurate data on the PPC platform we can safely conclude that the Webmaster Tools data is being filtered in some way. Google’s own blog says that this is
“aggregated data of the top 1,000 search queries that drove traffic…for the past 30 days”
so we can assume that this is still the case, even though greater granularity appears to now be visible in Webmaster Tools.
We did, however, find a lot of value in analysing the relative Click-Through-Rates (CTR) of the different search queries for each individual page. We found that in some cases the lower click-through could be attributed to the lack of that phrase appearing prominently on the page or in the meta description.
Armed with this knowledge we set about changing some of the pages for this client not least of which because we could now see which search phrases triggered the page being shown in search.
This is critical. It’s like keyword research only better.
Now we can see what real users are typing in and from this we can better shape the content on that page to match their search needs.
So, filtered data or not, it is a lot better than (not provided) in Analytics and the flexibility with which you can analyse the data is very useful.
Do you have any tricks in Webmaster Tools that you have found which are worth sharing? Feel free to add them to the comments below if you do.
