You've successfully subscribed to Thematic
Great! Next, complete checkout for full access to Thematic
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

How accurate is Thematic at analyzing customer feedback?

In discussions with potential clients, we always hear the question, “How accurate is Thematic?”

This question usually comes up for two reasons: 

“I’ve been doing the analysis myself. I don’t want to compromise the accuracy when handing this task over to an automated solution.”

“I realize that doing things manually isn’t the best approach. I tried different solutions. They all produced generic results that aren’t useful. Does a solution that’s sufficiently accurate even exist??”

Both are valid reasons, and we are eager to provide an answer.

But let’s step back, when it comes to analyzing feedback, what does “accurate” really mean?

Why is accuracy alone meaningless?

Having published dozens of academic publications that evaluate accuracy, I can geek out on this topic for hours. But, in the context of customer feedback, these three key points matter the most:

100% accuracy is a myth – and it’s subjective anyway 

Even people disagree when deciding how to code a piece of feedback. 

The level of agreement between people is called consistency, and it usually ranges between 40 and 70% depending on the task

So, don’t expect a generic number as an answer.

Usefulness trumps accuracy

Imagine this situation: All feedback theoretically falls into billing, price, or customer service

It’s relatively easy to be highly accurate, because there are only 3 choices.

But is the analysis useful? 

Unlikely, especially because you already knew these categories in advance. 

A solution with 100% accuracy that doesn’t get more specific than billing is actually worse than a solution that has an 85% level accuracy, but tells you which comments are saying “billing date is inconvenient” vs. “billing terms have improved” vs. “billing isn’t accurate.”  

Ultimately, it’s more important to get specific and actionable themes, rather than chasing an accuracy number. More on this later.

Ease and speed of refinement matters too

Refinement is a step that many people also overlook. 

No matter how “accurate” an approach might be, there are always human perspectives and tacit business knowledge that need to be incorporated in order to make the output actionable.

Oftentimes this knowledge and perspective only becomes apparent after a review of the first results. 

Let’s say we have evaluated two solutions:

Solution A is 80% accurate in finding themes. Its decisions are transparent, and anyone can easily refine them to make the results more actionable and relevant. Within a couple of hours you can get to nearly perfect accuracy.

Solution B is 90% accurate, but it’s a black box. It takes weeks of data science labour to refine the output. 

I know which one I’d rather choose…

How accurate is Thematic?

At Thematic, we’ve designed an interface that makes it easy to quickly refine themes, and have built algorithms that learn from this process, delivering improved results in future projects. 

Other solutions require extensive manual configuration, and with manual feedback analysis (which some might claim is the most accurate approach of all) refining themes can mean starting from scratch.

Previously, we’ve published a guide on how to evaluate analysis of customer feedback, and a white paper on the advanced evaluation of Thematic

In the white paper we used two different customer feedback datasets. Each one had themes from 4 people. 

We calculated people’s consistency with each other, and compared Thematic’s average consistency with them.

Our research shows that Thematic can easily be more accurate than people, whose analysis can suffer from personal bias or being tired.

Please note that Thematic offers an easy way of editing the themes to ensure highest accuracy possible. This process takes between 30min and 2h.

Try to quickly eyeball the themes that Thematic found on one of the datasets: “How can we improve things in our business school?”

The advanced evaluation results were as follows:

  • Pre-editing, Thematic was slightly less consistent than people. Post-editing, Thematic was better than 2 out of 4 people.
  • Thematic’s results were more specific because not only it discovered themes, but also sub-themes. Where people just provided the general categories, Thematic found both themes and subthemes.

How accurate is Thematic compared to NLP APIs?

In an earlier post on how to build a customer feedback solution in-house, we have tested the major players, IBM Watson, Google Cloud, Amazon Comprehend, on online airline reviews. 

The results were disappointing:

  • The keywords were generic and not useful. 
  • Only a small proportion is actually tagged: 32% in the best case. 
  • Even very simple variations like capitalization and plural/singular weren’t treated as the same thing. 
Left: IBM Watson – Right: Google Cloud
Amazon Comprehend

It didn’t make sense to compare Thematic’s accuracy to these major players. 

After all, these solutions have been developed on very different type of text: newspaper article, webpages, research papers. 

So while they might excel at those types of text, they fail on customer feedback.

How accurate is Thematic compared to other solutions?

Maurice FitzGerald has evaluated other publically available solutions on the same dataset as used in this article: DataCracker, WordyUp, Wordclouds.com, Sift Keatext, SurveyTagger, SPSS, Lexalytics, MeaningCloud, Text2Data and HP’s HavenOnDemand. His findings:

  • Most systems performed poorly because they can only extract the most relevant single word and people were by far better than any of the tools. 
  • Best performing systems were Lexalytics and HavenOnDemand. However they both missed Publish exam dates earlier, one of the most important improvement suggestions (which Thematic did find without manual intervention).

Why is Thematic so accurate?

Thematic’s proprietary solution was designed specifically for the analysis of customer feedback by world’s leading experts in this field. 

The research is lead by Thematic’s co-founder and CEO Dr. Alyona Medelyan who has a PhD in keyword extraction and more than 2500 academic citations in Natural Language Processing.

Accuracy is also ensured by allowing a person to refine Thematic’s choices: themes can be regrouped, renamed and deleted. 

This refinement is optional because even without help Thematic gets 80-90% of the way.

Thematic has also other advantages: It learns the use of language not just from millions of pieces of feedback it collects, but also from all the human refinements it has captured over the past 5 years. 

Best thing: It does not require pre-defined categories or annotated data. 

Takeaways

  • Solutions as accurate (if not more accurate) as people exist!
  • You don’t need to compromise the quality of analysis when automating this task.
  • Usefulness (specificity and actionability) matters more than accuracy alone.
  • Ease and speed of refinement matters too
  • Don’t just ask for accuracy, eyeball the results to get a sense of how useful and actionable are the themes.

At Thematic we care about accuracy a big deal! And we guarantee human-competitive results. After all, the CEO has a PhD in human-competitive thematic analysis!

Keep reading

Text Analytics Software – How to unlock the drivers behind your performance

Text Analytics Software – How to unlock the drivers behind your performance

Many organisations, large or small, gather customer feedback to improve their CX efforts and ultimately their bottom line. But gathering feedback alone can’t make much of a difference. We need to analyze our feedback to discover insights that inspire us to drive action at our organisations. Enter; Text

Melanie Disse
Melanie Disse
AI & NLP
Thematic Analysis Software: How It Works & Why You Need It (2020)

Thematic Analysis Software: How It Works & Why You Need It (2020)

Most likely, you landed in this blog because you have too much feedback to analyze. You sent out a survey or collected reviews or other form of free-text feedback. Now that you have this feedback in-hand, what do you do with it? How can you identify common themes in responses?

Alyona Medelyan PhD
Alyona Medelyan PhD
AI & NLP
3 best practices for coding open-ended questions

3 best practices for coding open-ended questions

Open-ended survey questions often provide the most useful insights, but if you are dealing with hundreds or thousands of answers, summarising them will give you the biggest headache. The answer lies in coding open-ended questions. This means assigning one or more categories (also called codes) to each response. But how

Nathan Holmberg
Nathan Holmberg
AI & NLP