The February 6 issue of InformationWeek had an article entitled, Credibility of Analysts. While I was pleased that this blog was quoted a couple of times, there were several issues that bear comment.

One was the insinuation that Gartner writes reports that are funded directly by tech vendors. Knowing how unethical that can be, Gartner simply doesn’t do white papers. Period.

InformationWeek also questioned the transparency of Gartner's research methodologies. In fact, Gartner has documented and published its research methodologies for all to see and has a team dedicated to maintaining and improving these every day.

Another comment may have left the impression that Gartner’s research is somehow influenced by Gartner’s investment in the SI Ventures II fund and by outside investors such as Silver Lake Partners. Gartner sold its investment in SI Ventures II last year. And a distinct line is maintained between the “research” side of Gartner and the “business” side (I’d include Silver Lake Partners here, but also Gartner’s relationships with vendor clients): Gartner’s analysts are trained very specifically how to steer well clear of that line when they join the firm. If someone inside or outside Gartner wants to raise a question about our independence, they are always encouraged to contact the office of the ombudsman.

Finally, the article questions why end users would pay for independent research when so much information is available free. This isn’t so much an objectivity issue as one of value. So, wearing my Gartner hat instead of my ombudsman’s, here’s my response: You get what you pay for. Gartner has 700 analysts studying all aspects of IT, using a rigorous, quality-controlled research process. Wouldn’t you rather have 700 full-time experts there to help you make mission-critical decisions worth millions of dollars that could make or break your business and career than depend on a document you got for free on the web?

Here are some links to previous ombudsman blog posts that address some of the article’s other issues:

"Pay to Play" – in addition to the fact that half of the vendors in our most recent "Cool Vendors" report are not clients, check out these posts:

Vendors can't "opt out" of a magic quadrant or pay to get in

In Response to Rob's 14 Sept. Comment - More on Pay-to-Play

Specific vendor client relationships:

Why Gartner Won't Release the Names of Vendor Clients

Transparency in evaluation models:

New Magic Quadrant processes: Have we done enough? You tell us.

I’ve also included below a couple of links that might be helpful for those with further questions:

Code of conduct that all Gartner associates follow.

To schedule a Vendor Briefing

Submit a Comment | Permalink

Hi Nancy and Larry,

The article in InformationWeek is rather bland and rehashes old issues, but your response and the commentary in the blogosphere (e.g., ARmadgeddon http://armadgeddon.blogspot...) is what is making this interesting.

I commend Gartner on the “no vendor sponsored whitepapers” policy. However, one of the reasons that there is this perception is that this policy was not always the case. If memory serves me right, it was not until the Forrester-Microsoft Linux whitepaper brouhaha (“Forrester alters 'integrity policy'” http://news.com.com/2100-10...) in October 2003 that Gartner changed its policy. Up to that point, Gartner Consulting – though not the analysts – did take on engagements writing vendor-sponsored whitepapers with input from the sponsor.

Cheers, -carter j

Carter Lusher
Director, Corporate Analyst Relations, HP

this blog is not a useful form of communication as now the posting are once every six weeks and no telling how much longer it would have been if you didn't need to post this defensive piece.

We changed our policy to prohibit white papers years ago, and yet there’s still a perception out there that we do them. Any idea why?

I'm going to answer my own question here: We changed our policy to prohibit white papers in August 2001 but didn’t widely publicize this. That is probably why the perception that we will do them persists.

I'm glad to see that you as Gartner's ombudsman (ombudspeople?) continue to address the eternal question of the objectivity and independence of analyst firms. And your response here to material on Gartner, specifically, in the recent InformationWeek article is good parry to the magazine's thrust.

But I think you do yourself a disservice starting at the point you switch to Gartner advocacy mode by saying "... wearing my Gartner hat instead of my ombudsman’s..." An ombudsman, while these days often an employee of the organization he/she is investigating, is supposed to be more detached than that. Refute errors with facts, but don't do marketing.

What is your position on Gartner co-presenting a webcast with a vendor? There are several examples of this at http://www.ebizq.com
Couldn't this be perceived as Gartner "supporting" a vendor?

Two questions:

1. Can you name a single instance in which a non-Gartner client ever made it to the leaders quadrant?

2. Can you name a single instance in which non-commercial open source ever made it to the leaders quadrant?

Answers to these two questions will uncover more in terms of integrity than anything else...

Alan, thank you for your feedback! You are correct that a Gartner analyst’s participation in a vendor-sponsored webinar could be perceived as Gartner supporting the vendor. To ensure that this is not the case, Gartner has strict rules about how analysts participate in these events, how the Gartner name is used, and how the vendor promotes the webcast. For example, Gartner analysts should always be listed as the “featured” speaker, and they can only present about general industry and market trends.

Gartner’s Vendor Relations explicitly approves all promotional material relating to the webcast, which uses the Gartner name or logo.

We have considered shutting down vendor-sponsored webcasts several times in the past, but the consistent feedback we get from end-user focus groups is that they get educational benefit from the presentations, and as long as the analyst is not promoting the vendor or products, the webcast is not an issue. We regularly re-assess our policies on these types of issues.

Additionally, we attempt to police all instances of our material being hosted on other sites, since without our express permission, these are violations of our policies. Thank you for bringing this to our attention … we will ensure that ebizQ will position our research in compliance with our rules.

As a former long-time Gartnerite and a current vendor client, I am happy that Gartner has created the Ombudsman post, and I'm confident that Gartner does better than most or all of its competitors in insuring the integrity of its research.

That said, I have a couple of observations to make:

1) A reader above, Chris Carter, submitted two questions on February 22 that you have yet to address. Hosting a blog with comments allowed is fraught with risk; not answering direct challenges reinforces the cynicism and snarkiness of those who are already predisposed to think negatively of you.

While I know that you cannot specifically say "Vendor X in MQ Y was in the Leaders quadrant, and we have never received a red cent from that company", you could select randomly 3 MQs from the last year in which such vendors have appeared as leaders and note that "Some vendors in the Leader quadrant are non-clients".

Silence in the face of criticism breeds contempt.

2) The Washingtom Post hires an Ombudsman as an outside contractor with a two year contract who must leave at the end of the term; this person has extensive editorial experience and no ties to the Post. He or she reports directly to the Board of Directors, and has no association with the newsroom. This person writes weekly, public-facing columns that take writers to task for lack of balance or for errors.

Are these the the terms you work under? You are both long-time veterans with close ties to the Research organization, and I have not seen you being openly critical of specific pieces of research.

What exactly is the role of the Ombudsman at Gartner if it is not to hold the Research organization (and specific analysts) to account?

Re: Alan's post about vendor-sponsored webcasts with Gartner analysts (and vendor marketing speaking tours with Gartner analysts) --

Perception is reality.

That perception is why vendor executives constantly say "Just throw a little more dollars Gartner's way" thinking that act will somehow shape analysts' opinion.

Just like with the elmination of vendor sponsored white-papers back in August 2001 (see Nancy's February 9th response to a "Credibility" comment http://ombudsman.blog.gartn...), Gartner would do itself a great favor by eliminating any activity that is directly sponsored by a single vendor -- and publicizing the policy change.

With respect to Chris Carter's comment from Feb 22 ... Yes, there are instances where a vendor that is not a client has been a leader in a Magic Quadrant. Client status just does not figure into this (or any) research. Gartner could not have survived for 26 years as a reputable, reliable advisor if vendors had to become clients to be selected as leaders. Likely, Gartner will never be able to erase everyone's suspicions short of no longer accepting revenue from vendors whatsoever ... and frankly, that probably wouldn't be sufficient either, given how Gartner's leadership position makes it a target for cries of foul. Let’s not demean the hard work that many vendors do to achieve a leadership position (and satisfy their customers) by asserting that they can just buy their way into that position.

Once again, I renew my past offer (http://blog.gartner.com/blo...) ... if someone can bring a situation to my or Nancy's attention where vendors are being forced to become clients to be covered or kept out of any quadrant (or any other piece of research) if they are not clients, we will completely respect your confidentiality, investigate the situation and make changes internally if necessary.

And just to be clear, while Nancy and I appreciate the open discussion on this blog, and our intent is to answer as many comments as possible, we never committed to answering every comment. Blog comments provide the ability for you to express your opinion. You (the commenters) can also increase our interest in answering comments by identifying yourself, at least with a valid email address. Your affiliation (whether this is a personal comment, you are a vendor, end-user or competitor) is important information for us and for the other readers of this blog, if you are to have any credibility.

Gartner modeled its ombudsman after that of the news industry, so your reference to the Washington Post’s ombudsman got me to thinking.

As we’ve been doing this for about a year-and-a-half now, I can say that while we designed our department after the news industry, the nature of the issues that have crossed our desks has dictated that the path we follow is not exactly the same as ombudsmen in the news industry (see my comments below).

To your point about the Washington Post ombudsman being a contractor with a maximum two-year tenure that reports to the Board of Directors: I think you’ll find that even news ombudsmen have very different tenures and contractual relationships with their companies. But one thing that is consistent is that their reporting relationship remains outside of either editorial or publishing departments. And this is true at Gartner, as well. I report to Gartner’s General Counsel, who operates completely independently from our business units, so we are not tied to any particular division. That was very important to us as we set this up and has proven helpful in practice.

To your point about Larry and me having deep roots in Research: Yes, Larry and I both have strong research backgrounds, but we’ve both been around Gartner long enough to know about the other parts of the business – and it’s a good thing, because more than half of the issues that cross our desks are not related to Research.

To your point about news ombudsmen spending their time taking writers to task for lack of balance or for errors: Honestly we haven’t had the luxury to poke around our pool of published research and report on what we don’t like to the public. We’re kept busy enough handling the issues brought to us and, when we get enough that are similar, analyzing root causes and making recommendations for remediation to our business units. (That said, we do find time occasionally to dig deeply into issues we simply have a hunch about.) But this is quite possibly the biggest difference between Gartner’s ombudsman office and one of the news industry, and perhaps what gives us some affiliation with government ombudsmen. We’re not “public editors” who proactively read Gartner research with the critical eye of the client and report on it in a weekly column. We react to issues regarding objectivity, balance and integrity; usually those are highly confidential and don’t warrant mention in a weekly column.

A last point on this: I do not mean to imply that Gartner is ridden with problems that Larry and I are tasked with channeling. Many of the issues we deal with aren’t problems at all, but rather sharing of best practices, requests for advice and planning for new products. These are what keep us sane!

One of my firm's former clients told me that they were a Leader in the Magic Quadrant that covered their industry, and that they had never been a Gartner client.

However, I do think there is an important factor here: Gartner's 'Ability to Execute' criteria correctly favor a supplier with revenue of $10 billion when they are up against a supplier with revenues of $10 million. Because Gartner focusses on selling to firms with revenues over $1 billion, larger organisations are more likely to be Gartner clients. Therefore, no-one cannot be surprised if Leaders are more likely to be Gartner clients -- but this does not suggest a causal relationship.

Neither it is the case that has Gartner stolidly enforced a 'glass ceiling' of revenue across all the quadrants, although some Quadrants might use revenue threasholds. For example, our former client was a longtime player in its market but has revenue under $100 million. At that time its revenue was quite possibly under $50 million. However, its vision and niche leadership gave it a secure place in its Leader quadrant despite that firm being both modestly sized and not a Gartner client.

Duncan Chapple,
Managing Director, Lighthouse Analyst Relations.

No response to Chris Carter's second question (2. Can you name a single instance in which non-commercial open source ever made it to the leaders quadrant?). Probably analysts are interested where the money is and not on open source.

I suggest your office provide more statisics.
Mopre information about what you are fixing up or suggesting. Gartner is a big organization so there will be stuff that you are fixing. let us know. the more you do that we all know about the better Gartner will be and the more perceptions will match reality.

chad
solution architect government