How to theme qualitative data using AI analysis software
If you ever had to analyze customer feedback, you will know that the most difficult part is to create a perfect code frame. You need to understand the dataset, the stakeholders involved and the ideal outcomes of the analysis. You will have to iterate before settling on a solution, which might still not feel perfect.
There are now many thematic analysis softwares out there that can help you manage this process. Thematic is one of them. Some rely on a lot of manual effort but automate theme management. Others use AI algorithms to automate parts of the process.
This amazing technology is changing what’s possible, but at the same time creates new challenges. How should you think about themes and topics? How to organize them? How does sentiment come in?
In this post we’re diving deep into theme editing and learning how to create the perfect code frame to analyze your qualitative feedback. You’ll learn what’s a theme, how to best organize them and how to combine themes with sentiment. We’ll also address how Thematic helps discover and organize themes, and how you can collaborate with an AI algorithm to get the best results.
How Thematic is different to other AI solutions
Thematic is an AI solution for analyzing customer feedback. Some AI solutions are designed as a black box: You only see the input and the output, but not what happens in between.
Thematic is different. It includes a special tool called Themes Editor. In this tool, users can see the intermediate outputs of AI. It has an intuitive interface that’s similar to how we organize files into folders. In the Themes Editor users can see why feedback was analyzed in a certain way.
This transparent approach means AI can be optimized even by those who aren’t data scientists. Human input makes the results more relevant and accurate. In the industry, this approach is sometimes called Augmented Intelligence, Human-in-the-Loop, or Human-Centric AI.
What is a theme?
Within the context of feedback analysis, a “theme” is the subject (or one of many subjects) within a piece of customer feedback. Themes are a core building block in qualitative data analysis. Themes meet the following criteria:
- Answer the question “what is this feedback about”?
- Mentioned by at least 2 people
- Capture meaning through a selection of keywords and key phrases
For example, the theme “ease of creating an account” includes key phrases like “to create an account” as well as sentiment keywords like “impossible”.
A keyword-based approach would match the user’s words exactly. But the theme-based approach captures the meaning regardless of how people might express a theme.
Thematic’s advanced AI groups comments in customer reviews into “themes” automatically. Thematic uses “Unsupervised Machine Learning” to discover synonyms, paraphrases and key phrases which are worth tracking. Thematic allows users to view these in the Themes Editor where they can review and edit mapped phrases, as well merging and unmerging themes.
What is a mapped phrase?
A mapped phrase is a keyword or key phrase that maps to a theme. This is unique to Thematic. For example, the key phrase “simplicity of account creation” falls under the overall theme of “ease of creating an account”. A theme can have between one and several hundreds of mapped phrases.
Some solutions require users to come up with such phrases, or to think up rules and patterns that capture them.
Thematic automatically discovers and suggests mapped phrases. In the Themes Editor, you can always add them manually or delete them. And you can also do it in bulk.
What is a merged theme?
A merged theme is a theme that is very similar in meaning to another theme. In the example below the theme “ease of registering” merges into the theme “ease of creating an account”.
Thematic automatically merges some themes. But you can always unmerge them, or merge other themes as necessary.
What’s the difference between a theme and a tag?
A theme captures the subject of a piece of feedback. A tag is a more generic way of capturing anything relevant to a piece of feedback, including a theme. For example, some teams create tags for specific features they launch. A customer might say “I cannot find my user name”, which could have a theme “unable to find username” and be tagged with “Customer Profile Upgrade”.
Once you have a set of themes, you want to tag all feedback with those themes. Tagging means attaching a theme to a piece of customer feedback. Tagging can be done by a person or by an algorithm. Thematic tags feedback with themes automatically.
By tagging all your feedback with themes, you are basically turning your qualitative data into quantitative. You can now start running all sorts of analyses and visualize qualitative feedback in charts, dashboards and reports.
At Thematic, we also attach a sentiment score to each tagged theme. Let’s dig into this in a bit more detail.
Combining your themes with sentiment
Themes can be linked to sentiment in customer feedback in two different ways. (A function of Thematic is the process of sentiment analysis.)
1. The theme expresses the sentiment, for example “difficult to create an account” or “easy to create an account’.
Themes expressing sentiment already capture meaning in a clear and expressive way. You can instantly take an action here, e.g. send everyone who submitted this feedback to your product team.
But these themes are not always a good choice for your code frame. Sometimes, when tagging feedback with a theme, the sentiment can be neutral or more specific than just positive or negative, e.g. “many features”, “professional staff”. If customers talk about many different attributes of a product or service in their feedback, having a negative and a positive theme will create an unwieldy code frame.
2. The theme can be neutral, for example “creating an account”, and sentiment is attached when a part of a sentence is tagged with that theme.
For example, in a sentence “It was easy for me in the end to create an account”, the theme “creating an account” has a positive sentiment. In a sentence, “It was impossible to follow the instructions to create an account”, it’s negative. The overall sentiment of a theme could be the average sentiment across all tags of that theme in feedback.
Handling sentiment in this way creates less themes and helps find less divisive themes. You can attach sentiment to any theme regardless of the additional sentiment or emotion it might carry, e.g. “exciting features”.
But this analysis can feel less actionable. For instance, imagine a piece of feedback “I struggled when trying to setup direct debit”. It can be themes in two ways:
- The theme is “Difficult to setup direct debit”.
- The theme is “Setting up direct debit” and the tagged sentiment is negative.
Both express the same information but the first theming feels more actionable.
Thematic combines both approaches, using the first approach for frequent themes and the second approach for infrequent themes. Just to make sure, it attaches sentiment to all tagged themes.
How to organize themes to fit into a hierarchy
To make it easier to work with themes, it’s best to organize them into a hierarchy. There can be two or three-level hierarchical code frames. Some use sentiment as an intermediate level in the hierarchy. But what’s the best way of organizing themes?
Let’s look at these two examples:
They organize a similar set of themes in two different ways. The code frame on the right seems useful at the first glance, because you could share all feedback that falls under “Difficult to use” with the UX team and they would have the breakdown by feature. But should they focus on improving the UX of registration or setting up direct debit? To understand which part of the UI is more difficult they need to compare the numbers in the “Easy to use” group first.
The approach on the left is focused on the functional areas of the product or service. Ease of use is a separate theme, which can appear alongside more functional themes. If we know that “Setup direct debit” has overall negative sentiment, we can now start looking into why. It may be that the UI is difficult to use or there might be errors outside of the UI.
Thematic is more likely to use the approach on the right when grouping themes automatically. It builds a hierarchy according to frequency of themes. If Billing is mentioned often (there is a lot of feedback tagged with that theme), it becomes a top-level theme, or a base theme as we call it. If a less frequent theme is similar to Billing, Thematic groups it underneath, it becomes a sub theme.
Working with themes to create an analysis
Now that you have a code frame of perfect themes and a way of capturing sentiment alongside themes, you can start analyzing your data. Here are some of the reports you can build in Excel, Tableau, Power BI or Thematic. If you can't decide between Tableau or Power BI, read our comprehensive guide: Power BI vs Tableau.
Themes by volume
This report is also called a histogram. It allows you to view the most common themes in the data. You can answer the questions like:
- “What do people talk the most about?”
- “Do users care more about X or Y”
- “What are the biggest differences between what matters to two customer segments?” (if you filter the data by customer segment first)
Here is how we create such a report in Thematic.
Themes by score
If you have a score, such as a star rating or an NPS score, you can group themes to see the ones linked to lowest and highest scores. They might not be frequent, but could be important.
- Themes linked to highest positive scores describe how you add value to your customers.
- Themes linked to the lowest negative scores describe where you fall below customer expectations.
Theme volume over time
If you have a time element in your data, you can visualize a trend. Is the frequency of a theme over time declining or increasing? This kind of report will tell you if customers noticed the improvements you are making in a specific area, or an issue that might be on the rise.
In this example, May-June 2019 saw a spike in feedback about depositing checks. It turns out that this was caused by problems related to the check depositing service of this app.
Thematic provides other visualizations out-of-the-box such as impact, waterfall and anomaly detection.
How to refine themes over time
As more feedback is added to your dataset and as your product and service evolve, your themes will change. You will want to reorganize the theme to account for these changes.
In Thematic, we built a tool called Themes Editor for managing your themes.
Our Themes Editor can also be used to review comments that don’t have any themes and run themes discoveries. There might be a new theme that can emerge from newly added feedback. The AI can also automatically suggest more similar themes to create granularity in one area of your feedback.