Max Diff Question Type

Max Diff Question Example

Using the Max Diff question type (aka maximum difference scaling) respondents are shown a set of the possible attributes and are asked to indicate the best and worst attributes (or most and least important, most and least appealing, etc.).

When to use it

Max Diff is an alternative to standard rating scale results that might lead you to believe everything is important. By forcing respondents to make choices between options, Max Diff delivers results that show the relative importance of the items being rated.

Consider the above example in which a respondent evaluates four support channels: Social Media Support, Email Support, Live Chat Support, and Phone Support. These four support channels create six possible comparisons

  • Social Media Support vs. Email Support
  • Social Media Support vs. Live Chat
  • Social Media Support vs. Phone Support
  • Phone Support vs. Email Support
  • Phone Support vs. Live Chat
  • Email Support vs Live Chat

If the respondent says that Phone Support is best and Social Media Support is worst, this informs us the outcome of five of the six possible comparisons. The only paired comparison that cannot be inferred is Email Support vs. Live Chat Support, because we do not know how the respondent ranks these two support channels in comparison to each other. As you can see, this produces better data than a standard ranking question. And people are much better at judging items at extremes than in discriminating the importance of items in the middle.

The Max Diff question type has the added benefit of showing random sets of attributes so respondents are not evaluating all attributes at the same time, which is really hard for people to do. And, by displaying the same attributes several times, you gather even more robust data.


What is an attribute? - An attribute is the property of the object, product, brand, service or advertisement that you are comparing.

What is a set? - A set is a randomly selected group of attributes.


Setup

  1. Click the Question link on the page where you would like to add your Max Diff question.
         Max Diff questions must be on a page by themselves on the Build tab as the respondent will need to use the next button to submit each set.
  2. Select Max Diff from the Question Type dropdown and enter the question you wish to ask.
  3. Under the MaxDiff Format section, select the Orientation you desire.
    Max Diff Orientation

         Best Practice Tip: Orientation

         We recommend the Worst | Item | Best layout option. We have found that this best conveys the task at hand. Survey respondents can easily comprehend that you would like for them to compare the attributes and select one for the each column. In addition, worst/least to best/most is a standard layout that most respondents will expect (this is for left-to-right languages).
    Max Diff Orientation Best Practices

  4. Next, you can also customize the Best and Worst Labels, as well as the Label for the Items respondents are evaluating.
    Max Diff Labels
  5. Next, scroll down and add the attributes you would like your respondents to evaluate.Max Diff Attributes
  6. Finally, above your attributes, customize the Number of attributes per set and the Number of sets per respondent. By default, your question will be setup to show 4 attributes and 6 sets. You will see the below warning as a result:

    Warning! With this combination of attributes per set and sets per respondent, not every attribute will be shown to each survey taker.

    We recommend setting up your question to at least show each attribute 3 to 5 times for each respondent. To do so, use one of the below equations; plug in the number of attributes (K) and number of attributes per set (k). 
  • 3 times per respondent: 3K/k = number of set to show
  • 4 times per respondent: 4K/k = number of set to show
  • 5 times per respondent: 5K/k = number of set to showMax Diff Warning Message

Survey Taking

On a desktop and most laptops, the Max Diff question type looks like the below. Remember, your Max Diff question should be the only question on the page!

Max Diff Survey Taking on Desktop

 Best Practice Tip: Add Instructions

It is a good idea to add instructions to the question text so respondents are aware they are evaluating different sets. We also recommend displaying no more than 5 attributes at one time. In addition, by displaying a couple of options in multiple sets you will get more robust data!

Best Practices: Max Diff Instructions

When optimized for mobile devices the Max Diff question type will convert your question to a mobile friendly format as pictured below.

By default, survey questions show one at a time on mobile devices to prevent the need for scrolling on smaller screens. You can turn off this one-at-a-time interaction if you wish.

Max Diff Survey Taking on Mobile

Reporting

Standard Report

We have improved the reporting of the Max Diff question in the Standard Report! It's much easier to understand and allows for more visibility into how items were ranked. For each attribute you will see the percentage of times it was ranked as most appealing, least appealing, or not chosen.

The attributes will be ranked based on the score which is computed using the below formula:

# times attribute was selected as best - # times attribute was selected as worst
____________________________________________________________________________
                                        # times the item appeared

From the score we can determine a couple of things:

  • The higher the score, the more the feature is appealing to respondents.
  • A positive score means that that attribute was selected as MOST appealing more often than least appealing.
  • A negative score means that that attribute was chosen as LEAST appealing more often than most appealing.
  • A score of zero means that that attribute was chosen as MOST and LEAST appealing an equal number of times OR it has never been chosen as most and least appealing.
  • If a score of an item is two times bigger than another item, it can be interpreted that it is twice as appealing.

Max Diff Reporting

 See additional compatible chart types

 See what other report types are compatible

The below grid shows whether Max Diff questions are compatible with each of our report types. If you plan to do some specific analysis within Alchemer this report compatibility chart should help you choose the right question types.

Report TypeCompatible
Standard 
Legacy Summary 
TURF

Profile

Crosstab

Comparison

Exporting

The Max Diff question has its own export which will be available under Results > Exports. Once there, select the Max Diff Raw Data option.

The Max Diff question does not export via the SPSS Export.

Max Diff Raw Data

In the export, there will be a row per combination displayed to the respondent. There will be a column per attribute displayed in the combination and a Best and Worst column. The reporting values of your Max Diff question will populate the rows. 

Looking below at Response ID 14, Combination 1 you can see that the following attributes were displayed to the respondent as their first combination: Help Documentation, Live Courses, Video Trainings, and Email Support. Live Courses was selected as best and Help Documentation was selected as worst. 

Max Diff Raw Data Example

FAQ

 Will each respondent see each attribute?

Not necessarily. The sets are selected somewhat randomly so it is possible for a respondent not to see every attribute in the list given the number of attributes, attributes per set, and number of sets displayed.

 How is the experiment set up?

The attributes are randomized. From this randomized list, a number of attributes are selected creating a unique set. This process is repeated creating all of the possible number of sets. It also randomizes the order that the attributes are displayed for each given set. (Ex: Respondent 1 may see ABCD and Respondent 10 may see CABD. These are the same four attributes, but they are presented in a different order.)

 Will sets be shown equally?

There is a check in place to look up how frequently a set has been shown to respondents. If a particular set has been shown more frequently than another set, it will be replaced with a less frequently viewed set.

 How many responses do I need? How many sets should I show? How many attributes per set should I show?

We're answering these questions together as the answers are functions of each other. There's no hard and fast formula for answering these questions but here are some guidelines:

  • Show four to five attributes per set. 
  • Show each attribute between 3 to 5 times on average per respondent across the sets.
    • The number of sets to display so that each item is shown on average 3 times per respondent is equal to: 3(K/k) where K is the total number of attributes and k is the number of items per set.
  • As far as number of responses, you generally need 200 response per segment* that you are targeting.

*This refers to segments in data analysis. If you plan to segment your max diff data, say by males and females, you will need the above number of responses for each segment.

If the attributes are displayed more than once in different sets, that will give you even better data. Some attributes could be ranked again against a different set of attributes, which can garner more information about its relative importance against a wider range of options.

Don't worry about whether each respondent sees all attributes. Instead, just collect a healthy number of responses; over a larger number of respondents all of the sets will be shown.

 Can I hide the x per y sets that displays below my Max Diff question?

You can! To do so, go to the Style tab of your survey and scroll to the bottom of the survey preview and click the link to the CSS/HTML Editor. Then paste the following code on the Custom CSS tab.

.sg-maxdiff-progress {
  display:none;
}

The Max Diff Question Type is not compatible with Page Repeating.

 

Net Promoter®, NPS®, NPS Prism®, and the NPS-related emoticons are registered trademarks of Bain & Company, Inc., Satmetrix Systems, Inc., and Fred Reichheld. Net Promoter Score℠ and Net Promoter System℠ are service marks of Bain & Company, Inc., Satmetrix Systems, Inc., and Fred Reichheld.

Basic Standard Market Research HR Professional Full Access Reporting
Free Individual SurveyGizmo Enterprise*
*Excludes Stakeholder
Feature Included In