Last September, MarketingSherpa, the online e-marketing publication based in Washington DC, released the first edition of the Buyer’s Guide to SEOP. It marked a first for our industry, as someone looked at SEO firms from an (arguably) objective viewpoint. For the most part, I think the guide is a valuable resource, and we recommend it highly to potential customers who are out shopping the market. That said, there are a few fundamental flaws in the methodology behind the rating criteria. I’ve certainly brought these to editor Anne Holland’s and researcher Anthony Muller’s attention, and I must say that MarketingSherpa has been quite receptive to industry and reader feedback.

The Need for the Guide

MarketingSherpa discovered there was a need for the guide after they ran a brief article in February, 2001 showing that some SEO firms might be overcharging for their services. This stirred what appeared to be a long simmering pot, as readers immediately responded with e-mails and phone calls asking what they should be spending and how to select a reputable firm.

MarketingSherpa discovered that this was a question more easily asked than answered. The SEO industry is in it’s fledging stages and many SEO firms are micro-businesses. Prices and quality of work are all over the map. Iprospect president Fredrick Marckini, one of the founders of the SEO industry, explains, “There are no barriers to entry. For a few hundred dollars, anyone can get a software package and set up shop as a SEO consultant. The leaders are starting to rise to the top, but there’s always a fresh crop of newcomers sprouting up.”

Now, this is not necessarily a bad thing. Some of these newcomers, with a little seasoning, will become excellent SEO providers. But, the fact is, right now SEO has gained an unfortunate reputation as a place where less ethical business people can make a quick buck. Because of this, MarketingSherpa felt the market was ready for a consumer’s guide that would take some of the guess work and risk out of the SEO selection process.

How the Guide was Researched

MarketingSherpa began by trying to find out just who the main SEO players were. They seeded online marketing forums and newsgroups with requests for recommendations of SEO firms. The list was compiled and run past Detlev Johnson, moderator of the iSearch SEO forum. Once the list was finalized at 27 firms for the initial edition, researcher Anthony Muller, a SEO consultant and president of Zenhits.com, began his research.

Muller began by creating a best practices questionnaire that would determine whether SEO firms indulged in some of the more questionable SEO practices, such as cloaking. Firms were awarded points based on their answers to these questions. A small amount of the overall score was also determined by a past ranking supplied by the SEO firm. Finally, Muller took a look at the example site to determine if what the firm said they did and what they actually did in practice were the same thing. From these three factors a letter grade, ranging from an A+ to an F was assigned to each firm. “Anything greater than a D indicates the firm produces acceptable work” explains Muller.

Broadening the Scope of the Guide

After the publication of the first edition, the industry rapidly made it the topic of heated discussion in many online forums. Many firms that weren’t included in the first edition (ours included) were scheduled for inclusion in upcoming editions. Firms from the UK and Australia were reviewed and featured in an edition released near the end of 2001, and more North American firms were included in late February, 2002, bringing the total up to 65. MarketingSherpa is currently working on the Spring edition, where 200 firms are going to be profiled. MarketingSherpa is still accepting nominations for the Summer edition.

It’s All About Rankings

Humans love rankings, and none more so than SEO professionals. Seeing where you rank is what we live and breathe every day. So, Mr Muller and the MarketingSherpa gang opened a worm can of epic proportions when they applied their ranking formula to the SEO industry. 

The very first release of the guide avoided rankings and simply listed the firms responses to the best practices questionnaire, as well as showing the sample client result achieved. Very soon afterwards, in response to readers requests, MarketingSherpa assigned scores for the right answers (or at least, right according to the editors) and a corresponding letter grade. Letter grades ranged from an A+ to an F. Predictably, firms that received lower grades were soon up in arms about their score, and weren’t shy about sharing their indignation with Anthony Muller. I met Anthony in Boston at a Search Engine Strategies Conference. The room was full of SEO consultants and Anthony commented, “I feel like I’m walking around with a bullseye on my back.” I think he was only half kidding.

How Many SEO Firms are Out There?

To show you how new our industry is, nobody knows how many SEO firms are out there. As I mentioned before, there are a lot of one and two person operations and there’s more springing up every day. MarketingSherpa’s guide has been the only attempt to catalogue the industry that has a high enough profile to encourage SEO firms to come out of the woods and ask to be included. In the upcoming Spring edition there will be profiles of over 200 firms, and MarketingSherpa is still accepting nominations for the Summer edition. My best guess is that there is somewhere between 350 and 400 established SEO firms worldwide.

The Plusses of Shopping with the Guide

Is the guide worth the money? Absolutely. If for not other reason than the researchers and editor have spent almost 400 hours talking to SEO firms about their practices and strategies. This exhaustive research has been compiled in a easy to read and compare format. The rating criteria may have some flaws, but it’s a quantum step forward from what was there before. I’ve looked at the guide in some detail, and when I compare the ratings on firms where I’m somewhat familiar with the quality of the work and their results, I find them reasonably accurate.

Simply put, this guide will save you hours of research, arm you with the right questions to ask and potentially save you thousands of dollars by steering you away from questionable SEO firms.

One Person’s Opinion

The biggest criticism of the guide seems to be that the best practices criteria is based on Anthony Muller’s opinion of what’s right and what’s not in SEO. While I happen to concur with Muller’s outlined best practices code, the overriding question is; Should what is becoming the defacto standards guidelines of the industry be determined by one person?

In the SEO industry, what is spam and what is not is ultimately decided by the search engines themselves. Each of the major engines has their own spam policy. To further complicate matters, what a search engine defines as spam and what they actively police against are two different things. Take cloaking for instance. Every major search engine has gone on record as calling cloaking spam, yet few of them (perhaps none) are actively searching out cloakers and deleting them from their indexes.

Suggestions have come from the search engine marketing industry that the search engines should work with optimizers to establish a universal best practices code or that a panel of well respected SEO professionals should be formed to police the industry and create a best practices code that could be used by consumers and publications like MarketingSherpa to determine who’s naughty and who’s nice.

There’s one other thorny issue to throw into the mix before I move on from this topic. SEO consultants are always coming up with new ways to tweak sites and bump them up a couple of notches on the search engines. Usually, these techniques are successful until they are widely used enough to attract the search engine’s attention. If the search engines determine that these new techniques are a deliberate attempt to mislead them, they may add them to their spam policy and actively police against them. But until then, these techniques can provide a strong competitive advantage to the SEO firm that develops them. If a best practices code is established, what happens when new techniques are discovered? Who decides if they’re spam or not. And what happens when a technique passes from being a brilliant SEO tactic to being spam?

More Hard Results Needed

Another flaw I see in the rating criteria put forth by MarketingSherpa and Anthony Muller is the weight put on the best practices score as compared to the weight put on the actual results achieved. In the first edition, only one result was used and it accounted for a small part of the overall score. Our business is all about results and they should form at least 50% of any rating criteria. And basing a ranking on just one result would make any statistician shiver in his boots.

In the second edition, MarketingSherpa has taken a first step by using three results rather than one. At this point, they haven’t said that the results will be weighted any higher in the overall score. I would like to see a minimum of 10 results used, spread over different engines and different clients. This would give a much truer picture of a SEO firm’s ability to deliver consistent results.

Secrecy about Criteria

MarketingSherpa should also be a little bit more forthcoming in the methods and formulas they use to rank the SEO firms. There is a general outline available of how they award points in the best practices section. But when it comes to rankings, they mention the factors (the actual ranking, the engine and the competitiveness of the term) but they don’t indicate how important each of these factors are. When SEO firms give them the results to be graded on, they are crossing their fingers and hoping that the results match what MarketingSherpa is looking for.

Anthony Muller has said that this is intentional, to prevent SEO firms from trying to artificially boost their scores by feeding the researchers what they want to see (hmmm..why does that sound familiar?). But in any type of competition or review process, the scoring criteria are known by all. This is like asking a football player to play without knowing the rules of the game.

A Need for Standards ..SEO poised for PrimeTime

What the publication of MarketingSherpa’s guide has done is generate enough controversy within the industry that several individual players have stepped forward and said that there is a need for standards that are universally recognized. The guide has served as a catalyst that may eventually result in the formation of a regulatory body. The SEO industry is in the throes of growing up, and along with that comes all the pains and controversy that every industry has to go through at the beginning.

Kudos for Mr Muller & Marketing Sherpa

There is something that has to be said here. It’s all to easy to offer criticism and heaven knows that a large quantity of it has been thrown both at Anthony Muller and MarketingSherpa. He has received hate e-mails and been named in lawsuits (for more on this, see Danny Sullivan's article), all generated from the part he played in trying to bring order to an industry that badly needed it. I hope the criticisms that I’ve offered fall in the category of constructive criticism.

The fact is that MarketingSherpa and especially Anthony Muller showed exceptional courage in undertaking this project. Anthony is a SEO consultant and he went in fully aware of the controversy that was sure to occur. In fact, he has chosen to discontinue his SEO business to avoid any conflict of interest that may arise through his association with the Guide. The SEO Buying Guide has been extremely good for our industry. In fact, it may be the single biggest step yet in bringing SEO out of the basement and making it a legitimate marketing alternative.

Buy the Guide!

The last part is easy.  If you're shopping for an SEO firm, buy the guide!