The rise and fall of open-ended questions

Recently I had an interesting discussion with Ron Stroeven, one the founders of Infotools, about open-enders, short for open-ended questions.

Infotools was established in 1990, but Ron has worked in market research far longer than that. He has a wealth of experience in survey design and data analysis, so it was fascinating to hear his view on open-enders, their history, and future.

Open-enders enable people to speak freely without being forced into predefined categories.

Nowadays a typical survey is done online, normally extending over a few days. It is a set of pre-structured multiple-choice questions, including scale questions. Sometimes one or two open-ended questions are added at the end as ‘throw-aways’ since the use of too many open-ended questions has traditionally been discouraged. This is because people’s answers to open-enders, known as verbatims, simply take too long to code and then analyze.

In the past, if the number of responses was manageable, open-ended answers were coded manually by a group of coders. It would typically take 6 to 12 weeks to run a survey, so coding responses manually as they arrived wasn’t much of an issue.

Coders would start by taking a sample of answers and creating a code frame. When new responses came in, the coders grouped them into this code frame. If new comments didn’t fit into any of the existing codes, they created new ones. Many agencies still do this to some extent!

Coding open-ended responses manually isn’t that practical anymore. Taking into account issues such as the ever-increasing and hard-to-manage code frames, human bias and inconsistency, automating coding seems to be the only practical solution. Thankfully there are now text analytics solutions that can help do just that.

These days, open-enders are on the rise, and here is why

People resist answering multi-page questionnaires with scale questions. The response rate is low and market researchers explain this with the term survey fatigue.

In order to increase the response rate, researchers try to keep their surveys short. More and more companies and organizations replace long surveys with the ultimate scale question: Would you recommend us to your friends and family?.

This generates the Net Promoter Score (or NPS), a single metric for measuring company performance in terms of Promoters, people who actively promote the company; Detractors, people who may actively detract; and Passives, who don’t do either.  

Customer loyalty and company revenue tie to this one metric, according to studies. Not surprisingly ‘NPS is the new black’ in customer experience and market research. Here is the Google Trends graph for NPS:

But how can you improve NPS?

The ultimate goal is to turn Passives (if not Detractors) into Promoters! How would one do this? By asking them what drives them to promote or detract.

Usually, there is the inclusion of two simple question types to each NPS survey: What impressed you? And What can we improve? People’s answers are gold.

Recently, the head of NPS at a large telco admitted to me: “If we want to improve the score, it’s all in the verbatims!”

So, the more people who answer NPS surveys, the more verbatims, or open-ended answers, there are to be analyzed.

Scaling up market research

Infotools know that text analytics can increase the use of open-enders. They can be used to create shorter questionnaires, replace sets of grid questions and enable people to speak freely without being forced into predefined categories.

Ron Stroeven says: "Market researchers have historically tried to control the range of answers through structured pre-coded questionnaires to facilitate the analyses. Now they instinctively try to control the respondent, even when it comes to open-ended questions."

"We have seen for example questions like What are the three things that impressed you? But what if there is only one real thing that impressed the respondent, or if there are five? Asking for three reasons distorts the accuracy of what people actually think, and detracts from the strength of open-enders. Text analytics makes this approach unnecessary, and there is no need to control what people express!"

Infotools believe that text analytics in surveys is a potential game changer. Getting to the open-end results faster or reducing cost is just part of the story.

The really exciting part is that companies can now add open-end questions to large-scale surveys. This has been otherwise prohibitive due to coding costs.

When using manual coding, scaling up surveys is linear: double the size, double the cost. When using text analytics in surveys, the gain is exponential. We are achieving the voice of the customer at a scale that hasn’t ever been possible before. That makes text analytics a huge value add for market research.

*Disclaimer: Infotools is a customer of Thematic.