You've successfully subscribed to Thematic
Great! Next, complete checkout for full access to Thematic
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

3 best practices for coding open-ended questions

Open-ended survey questions often provide the most useful insights, but if you are dealing with hundreds or thousands of answers, summarising them will give you the biggest headache. The answer lies in coding open-ended questions. This means assigning one or more categories (also called codes) to each response. But how to go about it?

Two Key Approaches to Coding Open-Ended Responses

Key Approach #1: Manual Coding

Manual coding has been invented many years ago in qualitative research. Here, a person codes each free-text response with one or more category. If you are dealing with hundreds or fewer responses, you can do it yourself, or you can hire an agency.

Read our guide on the manual coding of open-ended responses to a survey.

Key Approach #2: Automated Coding

Automated coding has become popular in recent years. This is an area of text analytics, where algorithms are used to do this task. Data scientists could use a generic NLP library, but tuning such libraries is hard, and the results are often difficult to interpret.

Solutions like Thematic use algorithms designed specifically for coding people’s responses. We recently published a white paper explaining the reliability of automated coding and ways of measuring it objectively.

3 Best Practices for Open-Ended Coding

Whether you go for the manual or automated approach, it’s a good idea to learn best practices from people who have been dealing with text for decades.

The main recommendations for open-ended coding are:

Fight Bias

Don’t start with a conceived notion of what you are hoping to find. Make sure categories or themes emerge from data rather than are pre-defined in advanced.

Provide Coverage

Make sure that the code frame covers a wide range of contrasting categories. For example, a technology company may have categories related to the UI of the product, customer service, the brand itself, stability, specific features and so on.

Ensure Consistency

If new categories emerge over time, all previously coded responses need to be re-analyzed. Keep iterating until each customer response is tagged with all themes present in the dataset.

Making Sense of Customer Comments in Surveys

How to use coding results to make sense of customer comments in surveys?

Either you will use an existing dashboard, or you create one yourself using Excel, Numbers or Tableau.

Here are the key things you need to capture:

  • What is the overall importance of each category in your code frame?
  • How do these changes depend on the segments of data?

The best chart to capture this is the old good bar chart.  Here is an example of what it could look like in a custom dashboard: 

Example of comparing themes of Promoters and Detractors
Example of comparing themes of Promoters and Detractors

And here is an example of how it could look in Excel or Numbers:

 

Comparing theme distribution for Promoters, Passives and Detractors using Numbers.
Comparing theme distribution for Promoters, Passives, and Detractors using Numbers.

Conclusion

Of course, what it looks like is not as important as whether the analysis is accurate and useful.

For more ideas, check out our article on effective Net Promoter Score analysis. The lessons will apply to other types of customer satisfaction surveys.

Start your free trial

Thematic is the easiest way to discover the best insights in feedback. Act on what matters to your customers and make an impact.

Keep reading
How to measure customer satisfaction: the complete guide
How to measure customer satisfaction: the complete guide

Everyone says they want customers to be satisfied, but what are you actually doing to make customers happy? How do you know if you’re on the right track? How do you know if your customer satisfaction efforts make a difference? Why even aim for customer satisfaction at all? We

Alyona Medelyan PhD
Alyona Medelyan PhD
Churn & Loyalty
How to super-charge your Qualtrics setup with Thematic & PowerBI
How to super-charge your Qualtrics setup with Thematic & PowerBI

Qualtrics is one of the most well-known and powerful Customer Feedback Management platforms. But even so, it has limitations. We recently hosted a live panel where data analysts from two well-known brands shared their experiences with Qualtrics, and how they extended this platform’s capabilities. Below, we’ll share the

Alyona Medelyan PhD
Alyona Medelyan PhD
Customer Experience
How we use our own platform and Chrome extension to centralize & analyze feedback
How we use our own platform and Chrome extension to centralize & analyze feedback

Customer feedback doesn't have all the answers. But it has critical insights for strategy and prioritization. Thematic is a B2B SaaS company. We aren't swimming in feedback. Every piece of feedback counts. Collecting and analyzing this feedback requires a different approach. We receive feedback from many places: our in-product NPS

Alyona Medelyan PhD
Alyona Medelyan PhD
Feedback Analysis