Gartner Research has made changes to the Magic Quadrant (MQ) processes - partly based on feedback from clients, and partly based on observations we have made about how MQs are used. We're excited about these changes, because they bring more transparency into the process, which frankly is important to all of the Ombudsman's constituencies: vendors, end users and Gartner associates. As you might imagine, we get more feedback on MQs than any other evaluation models we use - by far. Many of the changes we've made have been based on your -- shall we say "candid" -- feedback. Keep it coming.

You've told us that you want to understand more about what's actually behind the "dots" we place on MQs. Some typical questions:

• What criteria do you use, and how do you weight it?


• Why do some markets get MQs and others Market Scopes (MS), or nothing at all?


• Why do some MQ documents actually look different from others - are they truly different?


• Why do I have to toggle back and forth between MQ criteria and vendor documents; why can"t I just view it all together?


• Why don't you add the best features of the META Spectrum model to the MQ (for starters, providing relative weights to metrics)?


• Why can't you tell us even remotely when you'll be publishing MQs for the markets I care about?

We will answer these questions and more with the announcement.

We've also noted that some end-user clients have made limited - and often inappropriate - use of MQs by evaluating only the vendors in the Leaders' quadrant. You'll see in the updated MQ document that we explain why it's essential that end users not make such limited use of the MQ.

We think the changes are broad and substantive, but will they be enough to allow end users to derive greater value from our research? It's too soon to tell. Please let us know what you think about these changes as you see them unrolling. Did we convince you to look at vendors other than Leaders in a market? Have we provided enough consistency and transparency? If not, what should we do to change that? The more you tell us, and each other, what you think, the more we can continue to improve this process.

Submit a Comment | Permalink

I like much of the indicated change taking place. However, I don't see -- at this point -- indications that the new process/reports will address two MQ concerns that are quite common.

First, I'm well aware of an end-user perception that the MQs favor the largest -- and often most expensive -- vendors. Related to this is a vendor perception that the MQs favor vendors that are Gartner clients.

While the MQ changes sound good, I don't see how they address these issues head-on. For example, how about publishing correlations -- statistics comparing MQ rankings to quantitative industry ranking data (e.g. ranking by annual company revenues or company revenue growth)?

The vendor client status should be very straightforward to resolve. For example, how about indicating on MQs which vendors were Gartner research/consulting clients during the MQ compilation period?

I'd like to see Gartner prove that its analysts have no fear of revealing which vendors are their clients, and which are not. The MQ involves subjective analysis by human beings. Maybe your vendor clients score very well on completeness of vision criteria -- assuming they are adopting Gartner analyst advice on market definition, direction, drivers, product and service strategies, etc. --and maybe they don't. Would be good to put an end to the speculation.

I am greatly encouraged by this initiative as I have had many of the same issues with Gartners reports that have been identified. The only question I have is around "Offering (Product) Strategy" which is very closely related to "Ability to Execute" rather than "Completeness of Vision". In any case I am watching closely for the Service Desk/Service Support report to come out.

I am in the role of VP of Reserach Methodology at Gartner. Its great to get this feedback. I want to reply to Bill Black's comment to provide a little clarification. Using the Offering (Product or Service) Strategy criteria as part of the ‘Completeness of Vision’ analysis gives us the ability to look at the longer range plans for the offering and discuss how well they map to the overall direction and demands of the market over time. We balance this analysis with Product/Service criteria under ‘Ability to Execute’ that looks the issues related to effectiveness and match to the current market and client requirements.

These changes should help, but how are you holding analysts accountable to follow them? I have not seen any change in the process so far.

I agree with Barbara above that you should not fear disclosure of who are vendor clients and not disclosing it creates a lot of the criticism of the Magic Quadrant. Why don't you ask vendor clients if they would approve of having their status disclosed.

I would also suggest that you create a standard timeline for magic quandrant processes. You've identified the steps but how much advance notice should vendors be given? How much time should be allowed for survey completion? How long will the publishing process take? These should not be left to the individual analysts, who at times seem to create short timelines because they start the process leate, but should be set to reasonable timelines by Gartner management.

The Magic Quadrant process has many checks and balances to encourage analysts to follow the defined process. During each step of the review process with peers, manager and editors participants are looking for specific activities to have been completed. And of course, sharing the process with the vendors provides another check and balance because now the vendors have expectations of how the process will work. If the process is working differently than you expected please speak to the analyst about the process they are using. And, if you have concerns please don’t hesitate to contact me directly.

It isn’t possible for us to create a standard timeline for MQ and MarketScope processes because each project varies in terms of the type of research activities that will be done, the complexity of the analysis that must be completed and frankly the analyst’s schedule. However, we are hopeful that publishing the editorial calendar will help keep our research on schedule predictable