Search Engine Musings

If you can’t understand why your site isn’t getting Google rankings try checking your web hosting!

There are various reasons why it might be causing you to lose search positions. A few years ago there were reports that one large hosting company had been blocking Googlebot from spidering any of its clients’ sites, apparently in order to save bandwidth. That would certainly be a major breach of faith, but as ever the onus is on you as the buyer to beware. There’s some very cheap hosting deals out there, but you have to ask why they are cheap. Maybe they don’t have essential facilities or support, access to log files, database capability, scripting languages, or as in that case, maybe you’ll never get any rankings from them and therefore never be able to attract traffic. Free lunches generally have a catch somewhere.

Geographical considerations for search engines

How about checking where your host’s servers are. Search engines have various ways of establishing where your site is based and generally assume that where it’s based in where it’s aimed at. No problem if you have a .co.uk address - you’re obviously UK. Likewise, if your address finishes .fr .es or .it then you’re assumed to be French, Spanish or Italian. But what if you have a .com address? .com is something of a historical anomaly from the days when the net was in its infancy and everyone (well, everyone in America) thought you could just split sites into commercial (.com), non-profit (.org), educational (.edu), and government (.gov).

For a long time everyone jumping on the internet bandwagon assumed you had to have a .com address (the press even called it the .com bubble) but in fact the main problem with it is that it has no geographical significance. Without that major clue the search engines resort to secondary information, and one of those is where the servers are based. If you are a UK company with a .com address and your host’s servers are in another European country then you have a problem - Google and MSN/Live won’t include your site in their UK-only search results.

I was once phoned by a company who hosted their servers in Ireland (Eire) and complained they could never get rankings in the UK. Sure enough I checked their hosting and then checked Google and Yahoo in Ireland - they had top ten results all over the place. Unfortunately they didn’t have any potential customers there. In the UK where their market was they were nowhere to be seen!

The same problem has long caused difficulties for Australian webmasters who tend to go for hosting in the USA where it is apparently much cheaper. If they’re using a .com address then they won’t get results in their own country.

So watch where you host your site - it could make the difference between the success or failure of your business.

[edit - I must be psychic - the same day that I wrote this post the Google Webmaster Central Blog announced
Supplemental goes mainstream. Seems they don't agree with my reasoning and are determined to make it harder for us. Only time will tell if the results from the two indexes really do come together, but certainly the comments on that announcement seem pretty skeptical.]

There’s a lot of discussion at the moment about Google’s removal of a search method (though it looked more like a hack) which was supposed to allow you to check which of your pages were in the supplemental index rather than the main one.

It should be said that there was always some doubt about this method - it seems to have thrown up inconsistent results and there was a suggestion that it was possible to appear in both indexes at the same time so the results could be misleading. No wonder supplementals are a source of confusion.

At various times we’ve had Matt Cutts suggesting that we’re all too hung up on supplementals and even that Google might remove the supplemental indicator from the site: command results. [edit - which is what they've now done] The latter seems like a very retrograde step as it’ll leave people even more in the dark than ever about what Google thinks of their pages, and it will simply fuel rumours of the sort that cause so many problems for those without the experience to follow good practice or the money to employ professional help, rumours which Matt himself will then have to debunk.

So are we all hung up on supplementals? Well there seems little doubt that a supplemental page will not rank in a competitive field. I have recently seen a revamped site which had poor connection to its detail pages. To rememedy this an intermediate page was built which contained a great deal of information and linked to subpages which again had lots of good quality information around which the deepest detail pages were clustered. When this intermediate page first went live it immediately started to rank for a number of terms, and the subpages also started to rank. However a few weeks later the intermediate page dropped into the supplemental index, as did the subpages; the rankings immediately vanished, and have not returned.

Such a situation seems to lack natural justice and is likely to cause confusion amongst reputable webmasters who are trying to produce good sites - which is what Google claim to want. Since we don’t know exactly why pages can go supplemental some people will speculate and be tempted to take actions that may not be to the benefit of their users. Cutts has hinted that one of the reasons is lack of PageRank. That sounds dangerously like an invitation to point lots of links at the pages concerned as well as saying that quality is secondary to link strength. And then they wonder why people go out and buy links!

We can also look at the other end of the problem - what if we allow for the moment that there are perfectly good reasons in the algorithm for downgrading a page, and let’s say that a particular page isn’t good enough, isn’t worth ranking in the main index. Shouldn’t a conscientious webmaster be given a method by which they can check which pages of their site are regarded as poor, so they can take action to improve them?

Come on Google, give us the information we need to improve our sites, and then reward those improvements accordingly. Happier webmasters, better sites, better quality search results. Isn’t that what we all want?

Why should you care about Yahoo and MSN?

Increasingly I hear prospective clients say that they are only interested in Google results and nothing else. However, while Google are way out of sight in traffic referrals and both Yahoo and MSN/Live Search can be erratic and unreliable, I feel this is a bad approach to search engine optimisation in general.

There are two reasons for this:

• Over reliance on one source of traffic
• Narrow thinking in terms of site development

In the first case you have no fallback position if it all goes wrong. What if Google, in its attempts to eradicate spam, make another major algorithm change such as has happened a few times over the last few years? Even if you’re innocent you could still be caught up in the fallout and lose key rankings. That’s always going to hurt, but if you’ve no rankings elsewhere it could be fatal to your business. Rankings on Yahoo might just keep you going until Google works out the bugs.

The second case is to my mind even more important. If you’re constantly thinking about Google and what they’re doing then you’re not thinking about your customers. Your mindset may increasingly be to look for tricks and techniques that chase rankings rather than pursue good design, good content, and good usability. If that’s where you end up then you’ll find your conversion rate dropping because your customers no longer like your site.

SEO shouldn’t be about making poor sites rank above their level, it should be about building a site that deserves to rank well because it’s a good user experience, and then making sure it achieves that deserved ranking.

Yesterday word started filtering through from America of some fairly fundamental changes in the way Google display search results. This morning it hit the UK datacentres and we’re getting our first proper look at them. They look like being a massive change!

Google intend mixing into the main body of the search results the kind of results we’ve occasionally seen for local search in separate additional boxes at the top of pages. However these aren’t additional any more but form part of the normal 10 results (for anyone using the default setup). That’s not all, it wont just be local search; there may be book search, image search, video search, and others. Essentially Google has decide to include all the additional search features that have been sitting mostly unused on the menu bar above the search box into the main search results.

This change could have a profound impact on the usefulness of results, and depending on what you’re looking for it could be either much better or much worse. So far there is no sign of a way to avoid these changes in your preferences. Nor is there any indication of how often such results will appear.

For many companies who depend on first page/top ten results for traffic it could be a disaster if these new results push them off that vital first page. One example I’ve seen shows a search with local results where the three locals take up three places out of the ten. If you were in positions 8-10 the chances are you’ve been pushed down to the second page.

Only time will tell if this proves popular with users or not and what the ramifications will be for the SEO industry.
If users like it then it will cement Google’s position as the pre-eminent search engine and may consign the struggling MSN/Live Search to oblivion. However if users find their searches producing poor or irrelevant results then we might see a swing back towards anyone who can provide a good alternative. In the meantime I expect to see the conspiracy theorists out in force saying that it’s all a scheme to increase Adwords spending by the folk whose ranking get pushed down.

A few short years ago there was still a fair amount of competition for your questions and searches. Even when Google had become the major player there was still a good percentage of the market who used Yahoo and MSN, and a reasonable number who used Alltheweb, Hotbot, Alta Vista, and a few others, while the major ISPs and portals such as AOL, Netscape, Blueyonder etc. also had search results that were somewhat independent even if they were based on one of the major players.

Now however we are seeing almost total domination by Google and a lot of people are getting worried. A recent analysis of search engine share showed that both Yahoo and MSN/Live Search are down in single figures on the percentages and the rest are nowhere. Google UK alone beats all except Yahoo and is not far behind them. Google Canada alone has almost as much share as MSN. Google Germany alone beats Ask.

Meanwhile, reading the webmaster and seo forums shows a lot of people who are totally dependent on Google for their traffic and therefore their business survival and there is no longer an alternative strategy for them. They are worried, and with good reason. Even discounting the ones who proclaim innocense of unethical methods but are shown to be riddled with them when you look at their sites, there are still many who find themselves suddenly dropping out of sight when Google make a change to their algorithm. Indeed it happened to me when the last really major change took place about a year and a half ago - some terms that I had been clearly the most relevant site for dropped completely out of the results, yet once the BigDaddy update was resolved they suddenly reappeared as high as before and are still there. If I had been dependent on them for my income then their loss for 3 months could have had serious consequences. Yet throughout this time there was stony silence from Google other than the general platitudes about creating good content and many webmasters and small businesses will have spent a lot of wasted time making unnecessary changes to their sites in the vain hope of making a difference.

Such a situation makes people jittery and it hasn’t been helped at all by the recent utterances about paid links and suggestions from Google that you should snitch on your competitors. If this had been confined to the more blatant attempts to manipulate PageRank then that would have been one thing, but in fact they appeared to be declaring that all paid links are bad unless they carry nofollow tags, and that has really got a lot of backs up.

Suddenly Google, the company that everyone liked because of their simple, clean and advert-free interface, and their “do no evil” ethos that harked back to a more innocent and somewhat hippy-inspired internet, is being cast in the role of villain and their monopoly seen as a threat.

In fact the unthinkable has happened - Microsoft, the company that everyone loved to hate for their own monopolistic practices, is being seen as a last hope for competition and is being urged to follow through on the rumoured merger with Yahoo.

Google need to be very careful if they are to retain/regain their previous blissful public relations position. Trotting out cuddly Matt Cutts and cute Vanessa may not be enough any more. With worry about the ever increasing amount of private data being collected people naturally think about Big Brother (and I don’t mean the attrocious reality TV programme). They might just start thinking about George Orwell’s other well known line - “power corrupts, absolute power corrupts absolutely”. If that happens then instead of us worrying about how much Google trusts our websites, Google might have to start worrying about how much we trust them!

Since around August there has been an intermittent problem for .com sites with Google results in country specific listings (e.g. searching Google.co.uk and selecting UK-only results). What has been happening is that the home pages of these sites drop out of the country specific indexes, although they are still there in the global indexes. (To check if you’re affected simply do a site:www.mydomain.com in the two versions of the results) Along with this drop-out the rankings for any search term that is in any way reliant on the home page drop as well, and this has been causing problems for many businesses. A strange symptom that also often appears in these cases is that an https version of the domain address shows up in the indexes, even though there is usually no such version in existence.

There’s been a fair bit of discussion of this in the forums (for instance on webmasterworld) and those affected have reported that their sites periodically reappear and then drop out again, which agrees with what we’ve seen on some affected sites that we monitor. By no means all .com sites are affected and various theories have been tried and proved faulty as to what might be triggering these problems. The general opinion seems to be that at some point Google tried to introduce some sort of geo-specific filter which has gone wrong, and have subsequently been trying to fix it, however none of us really know and Google aren’t saying anything, even in reply to specific questions.

Sadly this is par for the course and is difficult to understand. We saw the same thing last year when what is now called the Big Daddy update took place. At that time many perfectly ethical sites dropped out of the indexes for a number of months while many spammy sites seemed to reappear. Eventually the update was rolled out and most of the good sites came back and reclaimed the rankings they had held previously. One of my own sites suffered this effect, even for some very specific terms for which I was clearly amongst the most relevant sites. At that time there was a great outcry but very little if any useful feedback from Google - just the usual generalities about making sure you do all the normal ethical things.

Now you can understand Google wanting to distance itself from making too many direct answers because they must certainly be inundated with queries from both ignorant users who can’t manage to read the T&Cs, and spammers who are trying to get back into the rankings after being banned. However when people have legitimate questions and there are provable problems with the results you would think that it would benefit them from a technical point of view as well as a PR point of view to be a little more communicative. While the sitemap / webmaster interface is a step in the right direction there are still many questions which are unanswered. SEO clients, who are often big businesses themselves, cannot understand how their SEOs can’t just phone up Google and get answers. The apparent climate of secrecy contributes to the “smoke and mirrors” reputation that SEO has.

The current situation only gives credence to the opinions of those cynics who point to the fact that these major index drops always seem to happen in the run-up to Christmas and are deliberately engineered to boost Adwords revenue. The best way to scotch such suggestions would be for Google to be more open about what is going on and admit any problems when they occur. Even if nothing can be done to speed up the solutions it would ease the minds of those affected, stop them tinkering with their sites in ways that may well be harmful in the long term, and greatly increase their respect for the company that was founded on the idea of “do no evil” but whose reputation is no longer seen as whiter than white.

Google aren’t the only ones to have some strange results at the moment. MSN / Windows Live is also doing some odd things. While checking my own listings I noticed that if you go to msn.com (which these days no longer redirects you to the uk version) and search for “search engine optimisation scotland”, the top ranked site is a holding page for a forthcoming site and contains no content apart from a handful of Google Adsense ads. Surely some mistake!