MarketingSherpa has recently released their new version of the Buyer’s Guide to Search Engine Optimization Services. This recent update differs from it’s predecessors in a number of regards. Primarily, the controversial “Best Practices” score has been dropped, and replaced by three separate scores, indicating client results, price ranges and services offered. Also, the driving force behind the guide is no longer the one who came up with the idea in the first place. SEO consultant Anthony Muller has been almost completely phased out of the project, to be replaced by MarketingSherpa’s new managing editor, Alexis Gutzman.

So Long Anthony, Hello Alexis

I asked Alexis the reason for Anthony’s departure. “The project has grown in size from the first edition,” replied Alexis. “We’re at 254 pages now. Anthony didn’t have the time to devote to the project. Besides, this is now a book. As a published author, I know what it takes to put a book of this size together. Anthony’s a great guy, but he’s an SEO consultant, not an author.”

Anthony Muller, on the other hand, had a slightly different version. “There were 2 different trains of thought and I didn’t want the real reason I created the report to get lost in profit margins and fear of liability. I was given very few choices in the matter.”

Whatever the reason for Muller’s departure, the fact remains that the focus of the publication has shifted. Muller’s intent was to draw a distinct line between the reputable and less ethical practitioners of search engine marketing. And, in Anthony’s New York style, there were few punches pulled. Under Gutzman’s direction, the clear delineation between good and bad has become fuzzier and the tone of the publication much more philosophical.

The Intended Target

Part of the reason for the change in tone comes from Gutzman’s perception of the needs of the intended audience. “This is a buyer’s guide for marketers. It’s not meant as an indictment of SEO practices. Policing the industry is not my job. I give the facts. It’s up to them to determine which is the best company for their needs.”

This approach is more in line with suggestions from critics of the first guide. Danny Sullivan, for one, was uncomfortable in labeling SEO practices as “good” or “bad”. Perhaps nothing tested this better than the practice of cloaking, reviled by Muller but treated in a much gentler way by the new guide. “When we asked the engines point blank about cloaking, most would not give us clear indication that it would result in being banned,” explains Gutzman.

Muller’s response, “Cloaking was seen as an unsafe tactic. I think many focused on it alone and lost the forest through the trees. However when the largest search engines stance is “All cloaking is Spam… period” and most other players say it is a “no-no with exceptions”, you can see why it was labeled as “unsafe”. This new edition of the guide had a shot at being less black and white, but I think it overcompensated a bit.”

Where’s the Ratings?

The most notable change is the dropping of the best practices grade. In the previous edition, this was a letter grade from F to A+ indicating whether the tactics used by the SEO firm were considered ethical according to Muller.  In the latest guide, there is no grade at all assigned for practices. Instead, the firm’s responses to questions about tactics used are published and separated into two categories: “Safe and Effective” and “Potentially Problematic”.

“I’m leaving it up to the buyer to decide what tactics they can live with”, says Gutzman. “The problem with the original best practices code was that it was based on the what the engines say, not what they do. Our interviews showed that there can be quite a difference between the two.”

One thing stands out when reading through the firm profiles in the new edition. After being caught the first time, firms were much more cautious in how they answered the best practices questions. Of the 55 firms profiled, only one admitted using cloaking on a regular basis. Other naughty tactics were pretty much unanimously disavowed by respondents.

This brings up another potential problem with the new edition. If you accept MarketingSherpa’s explanation that marketers are savvy enough to be able to distinguish between what’s ethical and what’s not in search engine optimization, you still have to have honest responses to these questions to make that call. If everybody is telling the marketers what they want to hear, the guide really doesn’t help in the selection of a reputable SEO firm.

Muller goes one step further, “Any SEM can “talk the talk! What they can’t hide is their actual work from the eyes of a SEM expert. I think this is the largest fatal flaw of the new guide. Without back end research, it can very quickly lull someone into a false sense of security and be worse than no guide at all.  What if someone reads the guide, hires one of the top ranked firms and then finds out that they’re spammers? What damage will that do to the credibility of the guide and the industry as a whole?”

Background Research

Here we reach the crux of the difference between the old guide and the new. Muller knew positioning intimately and could recognize the tricks used, both acceptable and not acceptable. The majority of Muller’s time on the first edition was spent in doing background checking on the work of the SEO firms involved and seeing if they were in fact doing what they said they were doing.

MarketingSherpa, as a cost saving measure, decided to take the SEO firms comments at face value. “We spent hours and hours checking SEO work in the first edition,” explained Gutzman, “and only found a couple of instances where the tactics used were different from what the firm said they did. We make a decision that the resources required weren’t worth it.”

“All the back end research for the new edition was finished before its publication.,” Muller points out. “I had found some really large amounts of Spam on certain SEM’s reviewed and brought it to Sherpa’s attention. The reasons they decided not to include it in the new edition eluded me. A few of these companies that had Spam discovered in their work appear to have very good scores in the second edition. I get nervous thinking about a marketer that chooses one of those.”

At this point, nobody involved with the buyer’s guide is a full time SEO consultant. Gutzman did some research on SEO and included a section on it in a previous book. Donna Pfledderer, who handled the majority of the US interviews, has never worked in the SEO industry. Of the three, Tim Gibbons, the UK researcher, has had the most experience, working with an online PR firm in London.

The Results Score

Another area where the guide came up a little short of what was promised was in the area of results ratings. The guide has said it would base the results score on three client results, rather than one as in the first edition. While the guide does show 3 results, the 5 star results ranking is only based on the most competitive of the 3. Why? Apparently respondents didn’t know what type of sample results to provide. “We saved SEO butt on this one!” laughed Gutzman, “A lot of firms would have done far worse if we based the ranking on all three results. They didn’t understand how we were scoring them. But we did want to show three results”

Gutzman also explains that this results score probably isn’t the best way to judge an SEO firm. “A lot of companies don’t need a 5 star company for the keywords they’re trying to get rankings for. A 3 or a 4 star company may be fine. There are other criteria to judge an SEO firm on.”

While I agree with Gutzman that more goes into the mix of what makes a good SEO firm than just rankings, I have to question the wisdom of downplaying the results. You buy a car to get you from point A to point B. It may have the best sound system in the world, but it still has to move. Good rankings are what buyers want in an SEO firm. Why would you purposely try to downplay an SEO firm’s ability to deliver them?

The Services Offered Score

A new score was added to the Second edition of the guide, and this marks a welcome addition to the guide. The services offered score offers a clear legend so buyers can tell at a glance what services an SEO firm offers. “We wanted to add a 3^rd dimension of appropriateness, so we added the specialties score,” said Gutzman.

Five areas of service are represented by 5 letters: O – Optimization, P – Pay-per-click management, M – Metrics (measuring performance and ROI), D – Direct Feed and I – International Submissions.

The Pricing Score

The Pricing Score survived pretty much in tact from the first edition. One to five dollar signs gives a quick indication of the average pricing of each firm. A word of caution here. This score tends to be based on the entry level package, and with some firms, the difference between the cost of the low end and high end packages is substantial. Buyers may pick a firm that’s shown as being quite reasonable, only to find that the actual quote they end up receiving is thousands of dollars higher.

Legal Protection

The first edition of the guide initiated a storm of protest from many SEO firms and a few lawsuits aimed at MarketingSherpa and Anthony Muller. It appears that MarketingSherpa learned to be much more cautious on the second go around.

In addition to dropping the controversial best practices code, it also made sure every SEO firm included had the chance to see the comments that would be published, make justifiable revisions and to drop out of the publication if it chose to. The firm had to sign off on the proof and return it to be included. Some declined to participate and so the number of North American firms represented actually declined from over 70 in the first to 55 in the second.

Comments Ad Nauseum

One section that may cause problems for MarketingSherpa down the road is the comments section. Sherpa published an online form that allowed visitors to submit comments about an SEO firm. From the beginning, Gutzman indicated that they intended to publish every comment that was submitted. So, for some firms, readers have to cull through dozens of obviously solicited comments from clients. 

But it’s the less glowing comments that could cause problems down the line. As far as I could gather, there are no safeguards preventing negative comments that could be defamatory. MarketingSherpa has no intentions of investigating comments to make sure they are in fact justified and true. They’re simply publishing them as is.

Bottom  Line

After reviewing the guide and talking to both Alexis and Anthony, it seems that the guide is currently a publication in search of direction. It’s still caught in the middle ground between Anthony’s initial vision and the profit driven realities of a publisher trying to make a dollar from it.

Alexis Gutzman explained that this publication is a Buyer’s Guide. Unfortunately, calling a publication a buyers guide implies that independent testing and research has taken place and that objective recommendations are given to help the buyer in their decision. From what I can see, this isn’t the case. What MarketingSherpa has done is gather the comments of 55 different firms and compiled them in one place. While this undoubtedly saves the prospective buyer some legwork, it doesn’t qualify as a buyer’s guide.

The SEO industry and the tactics used can be a confusing topic for someone who works in the industry every day. For a buyer being introduced to it for the first, assuming that they possess the experience to sort out good companies from bad based solely on face value comments with no background investigation is, at the very least, dangerous. As I mentioned previously, the first edition of the Buyer’s Guide was a huge, albeit flawed, step forward in the industry. In my opinion, the latest edition represents a step backwards from the first edition.