Search Engine Musings

The other day I was at an internet marketing meetup talk given by a fellow SEO and had time to reflect on the many changes that have occurred in the whole field of search and internet communications and marketing in the last 8 months since I was regularly blogging. The perspective of looking back over that period was both shocking and enlightening.

So where to start on a summary of changes. Even a brief consideration will give us the following:

• Recent Google layout changes and development

1. Further inclusion of video, images, news and local results
2. The Wonder Wheel
3. More description text
4. Rich snippets

• The rise of Twitter
• Increasing personalisation and localisation of Google search results
• Mobile search developments
• Google Chrome browser
• The canonical tag
• Wolfram Alpha search engine

Now of course in the last couple of days since I first started writing this post and got interrupted there have been two potentially game-changing events:

• Google are now able to follow some JavaScript links
• The way that no-followed links work with regard to PageRank flow has changed

One thing about SEO - it’s never dull or lacking a challenge! And to think some clients still read five year-old advice and then ask why you aren’t doing it.

I’ll be looking at each of these changes in the next few posts - assuming that nothing else even more important crops up in the meantime

An earlier post looked at why search position reports still have a place; although it should be said that this is changing rapidly as the search landscape changes. This one goes beyond that and looks at why rankings are only the first step; what you should be aiming for with your website and what you should be measuring to assess it. This post and others to come are aimed more at site owners rather than SEOs, who really should already know this stuff if they are looking beyond a very narrow definition of what SEO is. However it’s so easy to get bogged down in technicalities that it never does any harm to refresh your mind on why you’re busy working on projects whether it’s recoding sites or chasing links.

Profit - ultimately the only real measure that counts

At the end of the day any commercial company needs to make money and its website has to contribute to that. There are different ways, some direct and some indirect, in which it can do that. All of these ways depend on traffic.

The basic value of traffic

To some extent traffic alone can give you some revenue via advertising impressions. It can also contribute to your company’s PR and branding presence. How much value you place on that depends on you and your particular market and whether the site is purely standalone or part of a wider operation. (As an aside it always surprises me to hear businesses that spend thousands on Yellow Pages and other conventional print advertising, considering it well spent despite little direct evidence of return, who will dismiss the advertising value of their website and insist that they only count direct sales against it.)

Conversions

However the real value only starts to kick in when you have some sort of conversion. That could be a small contribution if it’s that the user clicked on a paid advert or affiliate link and left the site. Or it could be a subscription or a sale for a large value item. But it can take many other forms. It could be that it generates a lead that produces a profitable business to business collaboration - do you have measures to assign such a lead against your site?

It’s also perfectly possible for the money for a direct sale to be taken at another stage - your “conversion” may be that the user orders a catalogue or requests a callback which allows your sales team to interact with them as a likely prospect - so you have to define your conversions carefully and assign your successes accordingly or you’ll have no clear idea of what’s working for you and what isn’t.

Sources of Traffic

Assuming you’re running a commercial website (and few non-commercial sites would be spending money on detailed analysis unless perhaps they were a charitable organisation which had to justify it’s web spend) then your traffic will come from one of the following sources:

• Organic search results
• Paid search results
• referrals from others sites either by links and recommendations or through paid advertising
• direct visits from people who know your site from previous visits
• direct visits from people who’ve discovered your address via other channels such as TV or printed media.

Measuring Traffic

To get an idea of what is working on your site and to try to improve matters you need to measure your traffic and understand how it behaves in the site. There are many ways of measuring traffic, but unfortunately very few of them are accurate. This is an area I’ve studied extensively and specialised in for a while so I speak from direct experience.

Hit Counters

Back in the “old” days you’d see garishly styled hit-counters on many sites. These were generally based on a very simple bit of JavaScript and were hilariously inaccurate and easily fooled.

Online Stats Packages

Many web hosting packages come with free online traffic recording statistics which attempt to analyse the log files which should be recorded on your server - they’re usually worth exactly what you paid for them because most of them can’t differentiate between search engine spiders, site scrapers, and real human traffic. Even the better ones such as AWStats which give well-formatted multi-page reports are really only guessing at what’s happening. They can give some useful information in some areas but should be approached with caution and used more as an indication to further detailed study.

Logfile Analysis Programs

Then there are a range of reporting tools which run on your local PC to analyse those same server log files once you’ve downloaded them - programs such as Webtrends and SurfStats. Such programs can be much more detailed as there is no problem of load on the web server and they can be configured to some extent and updated when changes occur. However the nature of the logs means that in order to derive meaningful figures the designers need to make certain assumptions about what constitutes a page view or an unique visit, and there are complications due to such things as the caches used by the major ISPs. The result is that the major programs in this field seldom agree with each other and you can only compare reports derived from the same program.

Analytics Programs

These programs are the usual choice for the most detailed information - they all utilise some form of tracking system for each page of your site. This can be an embedded link to a small graphics file or a JavaScript program such as is used by Google Analytics. Commercial examples will usually have a little more detail in their reports but obviously have a direct cost. Free programs such as the one from Google are viewed on-line and thus of course dependent on their servers being available. You also have to consider that while the commercial ones will usually regard your data as confidential, using the free ones means you are sharing your data with the supplier in a much more open-ended manner.

Whichever of these analytics programs you choose there is much more data available than with the other systems and on the whole that data is more reliable, though it should be said that no analytics system is ever completely accurate.

You’ll be able to see the number of visits, unique visits, length of time on site and on individual pages. You’ll be able to see which keywords were used to arrive via organic search or pay per click. what the bounce rate was and you can usually set up “goals”, which can record activities such as making a purchase or requesting a callback or a brochure.

It’s these measures that are far more important than simple search rankings in determining how successfully your website is performing as well as giving clues to how to improve it.

Once you can read and interpret this type of data you’ll be in a much better position to understand how users react to your site and to draw conclusions about how you can improve it. For instance there may be a keyword phrase that you have a very high ranking for in Google. However the traffic that arrives from that source may be spending very little time on the target page and a large percentage may be bouncing off without looking at other pages. Close investigation of the analytics data may allow you to understand why and to take action to improve the page that the users are landing on.

A good SEO company should be able to not only achieve rankings but also improve your overall traffic, your understanding of how your site works, your conversion rate, and ultimately your bottom line.

If you’re experiencing an unexpected increase in bandwidth usage I recommend you take a look at your logfiles before celebrating higher traffic too much, you might find it’s all a Microsoft-induced mirage.

The other day I noticed one of my sites seemed to have been unusually busy and I checked the usage on the AWStats reports. Initially I noticed about 30Mb of hits from the Russian Federation, but soon realised that there was a much larger source of usage - nearly 1Gb of hits from MSNbot! That was actually twice the level of genuine traffic over the same period. Clearly something was amiss, and having read an article last week about the same robot apparently ignoring robots.txt files (MSNbot 2.0b is ignoring robots.txt and No Index meta tags) I went looking in the raw logfiles.

I soon found the problem - the bot was calling my blog feed about once every minute which meant about 56k every time. At that rate it adds up pretty quickly.  I first tried adding a line to the robots.txt file to restrict the frequency of bot requests but this had no effect. Next I tried blocking the blog feed directory but again the bot kept on requesting the feed. Eventually I was forced to try blocking msnbot from the entire site, and somewhat to my surprise it worked - I had been ready to use the htaccess file to deny the bot any access to the server at all.

Denying the bot for a while until Microsoft sort it out obviously means that I’ll lose rankings in MSN/Live, but the traffic from it is miniscule anyway compare to Google - 32 referrals compared to 828 for that site - so I can live with that if it means not getting hit for a Gb of pointless bandwidth in two weeks.

Get to know your logfiles - it could save you a lot of problems.

If anyone out there is still reading this blog I should offer some apologies for the long hiatus. A move of house following a marital breakup lead to a dramatically reduced amount of time to devote to any of my blogs including this one.

However it is now about time that I returned to offering regular opinions of SEO and the way the web is developing - there’s been a lot of changes in the last few months so hopefully I’ll get back in the saddle properly once I get back from an impending holiday. If you’re still here then thanks for your patience and I hope my future posts will be useful to you.

Following up on the second part of my last post there is increasing online debate about what SEOs should be reporting to their clients. Specifically the arguments surround search ranking position reports (or SERPS).

Some SEOs regard search position reports as pointless, pointing to the increasingly variable results depending on where you are searching from, whether you are logged into an account and thus getting personalised results, or even what you searched for earlier (whether logged in or not). They also point to the fact that search engine positions are not the end result, merely a step on the way to higher traffic, leading hopefully to higher conversions and sales.

On the other hand there are those who regard them as essential tools, and some whose clients insist on such reports, whether they understand the wider picture or not.

While in an ideal market it would be nice to have clients who are all educated in the fuller understanding of web marketing, I’m afraid that for the moment I have to side with the SEOs who provide ranking reports. What I always caution against however, is becoming fixated on them - obsessing about every little fall or rise.

Ranking reports are useful to me when I take on a new site, particularly one which is doing poorly. In conjunction with other assessments they allow me to draw inferences about the site and how it is being seen within its market. The changes in results that occur after I’ve made adjustments to the site can show me the response of the search engines to those adjustments and suggest further refinements. SERPS reports are also useful as a means of building a relationship with a client. For a new client who may be either entirely new to SEO or may have been burned by a previous company’s poor work, the ranking report provides reassurance that I know what I’m doing, that I’m prepared to demonstrate and explain results to them, and that progress is being made.

Now the market in the USA may well be different from that in the UK. Perhaps the corporate climate for internet marketing has matured sufficiently there that such reports are seen as unimportant compared to the improvements in traffic and sales. However over here, and certainly in the sector that I work in, there are still many people with only the faintest idea of what search engine optimisation is and how it can help them. They want to see tangible results that mean something to them. Even those with a better idea of the process may have very fixed ideas of what they expect - one of the problems I constantly come up against is clients who are happy to take my advice on “pure SEO” but don’t seem to recognise that my thoughts on layout, useability, and conversion optimisation are part of the whole web design/SEO/internet marketing mix. (Indeed there are some companies that specialise in only the areas of what might be termed “conversion optimisation”.) For these clients the process of sales conversion is not seen as being directly connected to SEO, so for them it would be pointless for me to try to use sales figures as an indication of SEO success because they have compartmentalised the processes. To them my job is to get them rankings, so rankings are the measure of success.

Naturally, as one who sees the processes more holistically, I try to educate clients to see every aspect of the business of running a website - design, coding, programming, SEO, marketing, conversion, etc. - as being inextricably connected. But sometimes we have to accept that online selling is a young industry and that many businesses have only recently become involved in it, and either have limited understanding or fixed ideas carried over from a bricks-and-mortar world. Given that I suggest rankings reports are here to stay for a little while longer yet.

A couple of months ago there were rumours from the USA of case sensitivity in Google ranking results. This week I saw the first evidence of it in the UK results for one of my clients. What I saw initially was fairly dramatic (I won’t show the actual keywords but just the pattern) on a query which consisted of two keywords and a placename:

keyword keyword place - position 22
keyword keyword Place - position 14
Keyword Keyword Place - position 124

The following day the variation was rather less severe but there were still noticeable differences, as indeed there were with other clients’ results. It may take a while for the variations to settle down and we see the real picture, but if this is going to be a permanent feature of the rankings then it has considerable consequences for both clients and SEO practitioners.

For one thing we need to try to work out what the average user would type in when making a search. Personally I’ve always used all lower case in the past, and I suspect many others do the same, but where a town or city or proper name is included some people (possibly older people) may automatically capitalise the first letter of that word only, whereas younger people used to text-speak may not. Then there are acronyms - would you search for “UEFA Cup results”, or “uefa cup results” when looking for European football scores?

Most SEOs agree that the content of headings within your text content helps tell the search engines what the following paragraphs are about and thus helps the page rank for the terms within it. Is your house style to use “Title Case For Headings” or “Title case for headings” or are you following the fashion for “title case for heading”? What if your preferred style clashes with the way people search so that you rank better for “Purple Widgets” but everyone searches for “purple widgets”? It could be we just opened a Pandoras Box and pulled out a minefield.

Another problem will be what to report to clients. Now by coincidence I followed a discussion on Sphinn this morning where a number of SEOs argued that we shouldn’t report ranking positions to clients at all - that it is traffic and sales that matter and that with personalised search, geographical biasing, and variations in datacentres it all varies too much anyway. That would be great, I’m all in favour of stressing the end result as the important factor, but in practice most clients are pretty much hung up on their ranking positions and follow them themselves in a rather unscientific and ad-hoc way all the time. Who hasn’t had a call from a client who sees a fall of a few places on a search term and immediately wants an explanation? (Which version of the search engine were you using, UK-only or world-wide results, were you logged into an account, what other searches did you run today, 10 results per page or 100, are you wearing blue socks or brown, (wouldn’t you just love to say that?!) etc. etc.)

The fact is many of them don’t understand SEO and are desperate for any sort of number to cling on to in order to be sure that they haven’t hired a snake oil salesman who will bleed them dry without any benefit to their bottom-line. They insist on reports. And if they don’t then their Managing Director does.

But you can’t report on all the variations of multi-word search phrases - for every three-word term you’d have to check at least five alternatives - as the extra overhead in doing so would be enormous and would probably get your IP address banned for doing too many queries or doing automated queries. So if case-sensitivity has come to stay we’ll have to discuss with the clients which terms are the ones that they want monitored. That’ll be fun! What’s the betting everyone stops reporting on MSN results!!

One thing about SEO, it’s never dull!