Search Engine Musings

And the wider issue of communicating with the search giant

For most of the last month or so there has been a problem for some sites in accessing the useful tools that Google makes available for site administrators. This manifested itself as a failure to verify the “ownership” of the sites using either of the two methods available, with an error message that varied between indicating a server timeout and a DNS error in looking up the site.

Unfortunately this wasn’t acknowledged as a problem until three weeks after it first started and at time of writing it still hasn’t been resolved on many of my own sites amongst many others. As a result many webmasters have been wasting time trying to solve non-existent problems with their sites and making pointless support calls to their hosting companies. At least in the last week there has started to be some individual responses from official Google staff to some of the postings on the relevant Google groups forum and this is a welcome development but it serves to highlight the fact that they are generally a very unresponsive company and getting hard facts out of them is extremely difficult.

Now to some degree I have some sympathy with their dilemma because if there were totally open channels of communication then they would be deluged with millions of queries and complaints, many of them half-baked or misinformed at best - we’ve all seen the nutters and chancers who complain bitterly about dropping rankings when their sites are riddled with blatant black-hat techniques and spam. However a way has to be found to allow genuine webmasters to report real problems.

With any system of the mind-boggling complexity of a global search engine there will inevitably be problems and bugs. But by not engaging with the webmaster community Google are missing a perfect opportunity to get exactly the sort of feedback that they need from people in a position to see the effects and give them early warnings of possible errors. No matter how good Google’s engineers are they aren’t looking at search results in the same intensive way that we are. Sometimes we’ll see puzzling inconsistencies in data that will ring bells for us, or we’ll see patterns when analysing SERPS results over an extended period. You can develop a sixth sense for when things are not quite right and this could be invaluable to them in tracing problems.

Remember the Big Daddy update? For months webmasters were baffled by perfectly good sites losing all ranking; of course there was a lot of noise from the less reputable as well but it was easy to tell that there were plenty of genuine people suffering. For quite some time the official line was that there was no problem and people should just clean up their sites and add more content. Of course many desperate webmasters ended up making major changes to try to get back some rankings and traffic to help their businesses survive. I myself lost a swathe of high rankings for things that I was clearly one of the most relevant sites for - not just dropped down a bit but dropped out of the index altogether - but was fortunately able to sit it out making no changes. A good while later we started seeing a particular datacentre with rankings that looked a lot like they should be, and then a few weeks after that all the datacentres had that data rolled out and all my top rankings returned. With better communications all that wasted effort, lost business, and vast quantity of forum chatter could have been avoided and maybe Google could have got enough useful feedback to roll out the corrected update a bit sooner. And they wouldn’t have lost so many friends and suffered such bad PR.

The development of Webmaster Tools was a great step forward but I’ve seen a number of oddities in it at times. For instance one client’s site was (quite naturally) largely based around two keywords yet one of them wasn’t listed in the “How does Googlebot see your site” section. This seemed bizarre since the same term was prominently listed in the link text pointing at the site, but it did raise suspicions about an apparent penalty they seemed to be suffering from when we took over their account. We tried emailing Google about it but received no response. That could have been an opportunity for a useful dialogue that would have helped us to ensure their site was clean and of good quality by knowing where to look.

Other issues come to mind. I regularly see a set of results coming round that is pretty obviously broken data - a range of ranking terms all drop out for a few days and then go back to normal for the next month or so only to repeat the cycle again. A couple of months ago the rankings for this very blog dropped away suddenly and I later discovered I’d been hacked at exactly the time the drop started. However I didn’t receive the message in the Tools that we are led to believe is sent in such cases. I was lucky and found the problem with the help of a correspondent. Others may not be so fortunate if they rely on the messages.

A feedback form of some sort within the Tools would be at least partly self-filtering of the nutters and they could be easily ignored in any case since their sites would likely be flagged up already as dodgy. Of course that wouldn’t help much in the current case since many of us still can’t validate our sites, unless it was situated at the opening page of the account.

So come on Google, let’s come up with a method of sensible collaboration that will help both sides.

Does valid code matter?

You often see that question on SEO forums, and the wide diversity of replies shows how much confusion there is on this. It usually degenerates pretty quickly into an argument between those who fervently believe valid code helps search rankings and those who don’t. This misses the point completely.

My opinion, confirmed by numerous examples of building sites that rank quickly and well, is that code should certainly be as valid as possible but that more importantly it should be elegant and it should be suitable.

What exactly do I mean by elegant? Einstein said that things should be as simple as possible but no simpler. Really well structured code is so simple that it can read almost as easily in the browser source code as it can when it’s rendered. That is the result of good planning at the design stage - not rigid planning that goes for a specific effect but flexible planning that can allow for changes requested by clients and future development of the site.

By suitable I mean coding and programming that has been designed to accomplish the tasks that the site requires while fitting in with the overall design. Not something that has been borrowed from other sites and which then needs hacks and inline styles to fit in.

Read your source code and if you see complexity ask yourself why it’s there and is it necessary.

It’s very rare that you need to have code that has classes or span tags on every paragraph or heading - they should inherit their styling from their parent elements. It’s extremely rare that you need to use tables for anything other than tabular data.

Complex dynamic sites provide plenty of opportunity to go wrong because very often such a site’s different modules will be built by different programmers, and if no-one has an overall vision for the site and controls the architecture of it you’ll often find that parts of its coding structure are derived programmatically using inline tricks which produce poor HTML. Being a good programmer doesn’t mean being a good web designer and if there are multiple people working on a site there needs to be good communications between them.

This is also where the SEO knowledge has to come in if it hasn’t already, because it’s rare for programmers to have any understanding of it. That’s not to criticise them - they only require it if they are taking on a wider role than just programming.

When validity matters

Note that since the second paragraph I haven’t mentioned validity at all, let alone as a goal in itself. Elegant code will nearly always be valid pretty much automatically. However code can be valid and not elegant. It’s perfectly possible to write complex and convoluted code that is perfectly valid but not at all appropriate.

There are many different types of errors that can cause invalid code. Some are of very little importance, some may be important for cross-browser compatibility purposes but not be show stoppers otherwise, while others may be fundamentally important to HTML structure. Search engines may well be able to ignore many of the less important errors but may be unable to interpret the more structural ones. For instance I have seen cases where crucially placed duplicate IDs have caused spiders not to follow navigation systems. One site appeared perfectly ok in all other aspects and no-one could understand why it wasn’t being properly indexed until I pointed out this problem and corrected it; whereupon the missing pages suddenly began to appear in the indexes.

You won’t get prizes from the search engines for just having valid code but you will get benefits from well-designed, simple and elegant code; and that’s what you should be aiming for as both a web designer and an SEO.

There seems to be a lot of confusion in the minds of many webmasters, both new and experienced, about what SEO actually is and what tasks it is useful to spend time on. You see forum questions asking ’should I worry about code validity’ (I’ll return to that one in another posting soon), ’should I put in a better shopping cart system?’, ’should I worry about Firefox/Macs/disabled people?’. You soon get the impression that these webmasters aren’t looking at their web sites with any sort of rounded view.

A web site isn’t there just to get rankings, just to get traffic, just to sell stuff to IE users. You have to look at it as a whole - how do the various parts fit together to attract users, attract genuine links, satisfy the demand for the product, service or information you provide, abide by the law, allow search engines easy access, and a dozen other areas.

It’s not just that SEO and usability go together - everything about a web site should be optimised to make visiting it a quality experience. Navigation should assist users to find what they want. Content should be laid out and structured to be easily scanned while providing maximum information. Images should be sensibly sized and add to the information rather than just acting as generic filler. If you’re selling something then it should be possible to order it with the minimum of fuss and with as many payment options as possible.

There are no shortcuts to quality. Think total web site optimisation and you won’t go far wrong.
To me that’s what true SEO means, but the terminology in our industry is so fluid that it can be taken to mean just about anything and many people see it in much narrower terms. If you’re hiring an SEO company then find out beforehand what they think it means and define how wide their remit should be.

Some of you will by now have seen the new BBC home page with its movable and customisable sections which can be tailored to your preferences. Fiddleability rules!

When I first looked at the beta version I wasn’t convinced; it looked a bit “play-school” and some of the option weren’t too well thought out. But it’s now much better and if you reduce the text size a notch it fits together pretty well to give you a selection of news items of the sort you want without the stuff you dont.

I can see this becoming the defalt home page for a lot of people, and maybe not just those in the UK either!  Depite the seemingly universal dumbing down all over the media the public still trust the BBC in a way that applies to very few other organisations. So the question is, will this take some traffic away from Google?

I suspect it might, particularly if the search facility was made more prominent and the “All the Web” option promoted a bit more. It’s ages since I tried any competitive searches on this engine - in fact I’d forgotten if it was still their own engine or if they were using someone else’s results - but I tried a few queries today and was quite pleased with the results. (they must be good, I was in the top ten for “search engine optimisation scotland” ;-)  ) If I have time I’ll try and remember to do a few more and compare them to Google and Yahoo.

If the BBC really wanted to push this I suspect they could get a reasonable share of the market. Who do you trust more - MSN, Ask, or good old Auntie Beeb? Well, that’s third place pretty well assured then! Of course what they would be allowed to do with it is another matter; they can hardly start up a BBC adwords-style operation can they?

Earlier this year I wrote about the rankings that this blog had enjoyed dropping substantially despite the main site holding on to its positions. Yesterday I discovered a possible cause. A correspondent on one of my other blogs notified me that his anti-virus program had alerted him to an attempted trojan link when he visited my other site. I investigated and found that a section of JavaScript had been added to my header.php file which used character code to open an iframe containing a link to a malware site. I removed it and restored the original clean file. Naturally I then checked my other blogs and discovered the same problem on this one. (Interestingly another blog based on a different template was clean, though that may just be a coincidence.)

The dates on the infected header.php files were the 18th and 19th of January and I don’t know yet how the JavaScript code was attached, but I’ll be updating the Wordpress installation (ironically I hadn’t upgraded immediately to 2.3 because I thought 2.1 was stable and secure) and trying out a new security technique that I discovered last night. If you have visited the site since those dates then I’d advise you to run your anti-virus programs. I’ve checked my own machine and found no problems so my own security seems to have held firm.

Now for the SEO implications. I check some of my rankings every week. On the 17th Jan they were fine with a number of top 5 results. By the next check on 24th Jan they had dropped substantially in Google. This strongly suggests that Google had identified the malware link and marked the blog site down for it. The blog rankings continued to tumble, suggesting that each fresh visit from Googlebot was pushing it further down. However I haven’t had any messages in Webmaster Tools notifying me of any problems. The question now is whether I should wait and see what happens after the next couple of Googlebot visits or if I should send a reinclusion request straight away. Suggestions welcome!

One thing for sure. I’ll be checking the file dates on my blog files regularly and keeping an eye out for any JavaScript in the source code.

Be careful out there!

Interesting; midday on Saturday (UK time) and Google is down. A tracert command runs fine on both the .co.uk and .com addresses but the wewbsite doesn’t respond. Not only that but sites that I know run Google Analytics are slow to load, presumably because the analytics calls aren’t responding.

My first thought was to look for news on this but what do you do when your main source of news is the thing that’s not working? Nothing on the BBC News (probably don’t think that new-fangled internet thing is very important), nothing on the various blogs and RSS feeds I monitor. Let’s try Yahoo - nothing useful in their web search (stuff from an outage in 2005), and their link to news takes ages to load before again showing nothing current. How about MSN/Live - hmm, taking ages to load and then eventually appears with no CSS formatting or images. The search box is still there though - but nothing in the results. The alternatives aren’t doing too well are they.

Checked three SEO forums but nothing there; guess the Americans are all still asleep. Some UK blogs, nope nothing there either. Now in the old days we’d have had Fidonet…

If it wasn’t for the single hop tracert result I might wonder if it was an ISP problem. Hmm, just noticed the Sphinn feed isn’t responding either. I’m beginning to feel as if I’ve woken up in a parallel universe or a sci-fi story and people are going to reply, “Google? what’s that?” Like Beverley in ST-TNG when she’s trapped inside a collapsing warp field and people keep disappearing.

So, this all begs the question. What do you do when the company that has almost a monopoly on information disappears. Expect further thoughts on this. (Though you might not be able to find them if Google doesn’t come back up!)

(Edited to add that Google came back up around 3.45pm )