You've successfully subscribed to Thematic
Great! Next, complete checkout for full access to Thematic
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

The rise and fall of open-ended questions

Recently I had an interesting discussion with Ron Stroeven, one the founders of Infotools, about open-enders, short for open-ended questions. In 1990 Infotools was established, but Ron has worked in market research far longer than that. He has a wealth of experience in survey design and data analysis, so it was fascinating to hear his view on open-enders, their history, and future.

Open-enders enable people to speak freely without being forced into predefined categories.

Nowadays a typical survey is done online, normally extending over a few days. It is a set of pre-structured multiple-choice questions, including scale questions. Sometimes one or two open-ended questions are added at the end as ‘throw-aways’ since the use of too many open-ended questions has traditionally been discouraged. This is because people’s answers to open-enders, known as verbatims, simply take too long to code and then analyse.

In the past, if the number of responses was manageable, open-ended answers were coded manually by a group of coders. It would typically take 6 to 12 weeks to run a survey, so coding responses manually as they arrived wasn’t much of an issue. Coders would start by taking a sample of answers and create a code frame. When new responses came in, the coders grouped them into this code frame. If new comments didn’t fit into any of the existing codes, they created new ones. Many agencies still do this to some extent!

Coding open-ended responses manually isn’t that practical anymore. Taking into account issues such as the ever-increasing and hard-to-manage code frames, human bias and inconsistency, automating coding seems to be the only practical solution. Thankfully there are now text analytics solutions that can help do just that.

These days, open-enders are on the rise, and here is why

People resist answering multi-page questionnaires with scale questions. The response rate is low and market researchers explain this with the term survey fatigue. In order to increase the response rate, researchers try to keep their surveys short. More and more companies and organizations replace long surveys with the ultimate scale question: Would you recommend us to your friends and family?. This generates the Net Promoter Score (or NPS), a single metric for measuring company performance in terms of Promoters, people who actively promote the company; Detractors, people who may actively detract; and Passives, who don’t do either.  Customer loyalty and company revenue tie to this one metric, according to studies. Not surprisingly ‘NPS is the new black’ in customer experience and market research. Here is the Google Trends graph for “net promoter score”:

But how can you improve NPS?

The ultimate goal is to turn Passives (if not Detractors) into Promoters! How would one do this? By asking them what drives them to promote or detract. Usually, there is the inclusion of two simple question types to each NPS survey: What impressed you? And What can we improve? People’s answers are gold. Recently, the head of NPS at a large telco admitted to me: “If we want to improve the score, it’s all in the verbatims!”. So, the more people who answer NPS surveys, the more verbatims, or open-ended answers, there are to be analyzed.

Scaling up market research

Infotools know that text analytics can increase the use of open-enders. They can be used to create shorter questionnaires, replace sets of grid questions and enable people to speak freely without being forced into predefined categories. Ron Stroeven says: “Market researchers have historically tried to control the range of answers through structured pre-coded questionnaires to facilitate the analyses. Now they instinctively try to control the respondent, even when it comes to open-ended questions. We have seen for example questions like What are the three things that impressed you? But what if there is only one real thing that impressed the respondent, or if there are five? Asking for three reasons distorts the accuracy of what people actually think, and detracts from the strength of open-enders. Text analytics makes this approach unnecessary, and there is no need to control what people express!”

Infotools believe that text analytics in surveys is a potential game changer. Getting to the open-end results faster or reducing cost is just part of the story. The really exciting part is that companies can now add open-end questions to large-scale surveys. This has been otherwise prohibitive due to coding costs. When using manual coding, scaling up surveys is linear: double the size, double the cost. When using text analytics in surveys, the gain is exponential. We are achieving the voice of the customer at a scale that hasn’t ever been possible before. That makes text analytics a huge value add for market research.

*Disclaimer: Infotools is a customer of Thematic


Keep reading

Text Analytics Software – How to unlock the drivers behind your performance

Text Analytics Software – How to unlock the drivers behind your performance

Many organisations, large or small, gather customer feedback to improve their CX efforts and ultimately their bottom line. But gathering feedback alone can’t make much of a difference. We need to analyze our feedback to discover insights that inspire us to drive action at our organisations. Enter; Text

Melanie Disse
Melanie Disse
AI & NLP
Thematic Analysis Software: How It Works & Why You Need It

Thematic Analysis Software: How It Works & Why You Need It

You know you need to collect customer feedback. You send out surveys and read reviews. But once you have customer feedback in-hand, what do you do with it? How do you identify common themes in customer responses — and turn that into actionable business insights?   You could spend long

Steven Plaat
Steven Plaat
AI & NLP
10 insider customer experience tips according to Shep Hyken

10 insider customer experience tips according to Shep Hyken

Shep Hyken knows a thing or two about customer experience. He needs no introduction in industry-wide circles, but in case you’re not familiar with his name: Shep is an award-winning customer service and customer experience speaker, a New York Times and Wall Street Journal best-selling author, A.K.A

Agi Marx
Agi Marx
Churn & Loyalty