You've successfully subscribed to Thematic
Great! Next, complete checkout for full access to Thematic
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

Why Thematic is superior to OpenAI for analysing feedback

With easy access to OpenAI's ChatGPT product, organizations are exploring its abilities. Can they use it as a tool to quickly analyze data sets, such as survey responses and customer feedback?

Yes, OpenAI is a powerful tool. It is useful to analyze smaller datasets and to get a quick overview. However, it is not capable of the accurate and detailed analysis that can be achieved with specialized feedback analysis tools.

We had a chat with Thematic CTO Nathan Holmberg and Rob Dumbleton, Head of Data Science & Research to understand this more. We talked about how to use ChatGPT to analyze a dataset of feedback comments, how it can help and what to watch out for. Nathan and Rob also shared how to overcome the limitations of OpenAI in analyzing feedback with Thematic.

Let's delve into what Nathan and Rob discussed:

First, what are we talking about when we say ‘OpenAI’?

OpenAI has become a catch-all term for new generative AI models and approaches.

OpenAI is the parent company, while their products bundle their generative AI into an interface where you engage with the large language model (LLM); ChatGPT is the model that most people are now familiar with. ChatGPT3.5 is the free, most commonly used chat interface, while ChatGPT4 is a more advanced version with a larger (and more expensive) underlying model.

Anyone who has experimented with ChatGPT, particularly 3.5, will know that it still comes with the potential for frequent inaccuracy and hallucination. While ChatGPT4 offers improved accuracy, the downside of this larger model is slower processing - around twice the time it would take on the smaller model.

Understanding OpenAI’s strengths and weaknesses

ChatGPT has its place for doing simple analysis of a limited data set. However Nathan and Rob shared that it does come with some clear limitations.

  • ChatGPT is clever but lacks the insight you need

ChatGPT excels at generating human-like text based on the input it receives. It can understand context, generate creative content, and even engage in meaningful conversations. However, when it comes to feedback analysis, OpenAI's capabilities are broad and general, lacking the specific focus required for in-depth insights into user sentiment and preferences.

  • Get a quick analysis (but don’t expect it to be reliable)

ChatGPT’s prerogative is to give you a satisfying answer, but with feedback analysis, what we’re really looking for is an accurate one; we need a true picture of customer needs.

ChatGPT becomes problematic when it draws on its training data, rather than solely extracting from the data you’ve provided it. This means responses can be questionable in their accuracy and their consistency.

Good, clear prompts help you to correctly filter the data and rely on it for relevancy, but finding the right prompt for every question is an inefficient way to manage the analysis process. Particularly when you have a large dataset and many questions to answer.

  • Helpful for small, simple data samples

It’s super simple to take comments from a survey or reviews, input it into ChatGPT and ask it to provide you with general sentiment based on those responses. For example, you might ask, "What’s the biggest concern for our clients?”. This works well for initial exploratory research if you’re working with 100 or fewer responses, where you can also take the time to cross-check validity yourself. It will give you a good feel for overall sentiment, but not an overly accurate report.

  • A longer way round to the answer you need

ChatGPT’s reliability drops as your dataset grows, and it becomes unrealistic to have to read and validate its analysis for accuracy yourself. You can also prompt ChatGPT to provide you with evidence. For example, if you are extracting themes from responses, you can ask Chat GPT to show the relevant data it used to reach its conclusions. However, again, it’s not the most efficient way to cross-check accuracy.

  • Good for one-time inputs

ChatGPT has also proven ineffective when it comes to tracking responses over time, comparing time periods or drilling down into quantifiable details, for example asking it to analyze and compare data across different regions. ChatGPT will struggle if you have to input data across multiple prompts or need to add new data sets, and can’t segment by specific criteria or responses.

What can Thematic do that ChatGPT can’t?

As insights professionals, your role is to accurately represent and report on the voice of your customer. If analysis isn’t accurate, and insights can’t be trusted, your customer's voice will essentially be ignored - the very opposite of what you’re trying to achieve. That’s why insights reporting should be precise, reliable and consistent in the type of data they draw on.

💡
"AI offers powerful opportunities to organisations by enabling insights to be surfaced from all the interactions with their customers.

If stakeholders detect inaccuracy or inconsistency in the analysis, they will ignore the insights outright. So AI insights must build confidence and trust, and be able to be traced back to the original interaction."


Robert Dumbleton, Head of Data Science and Research, Thematic

Insight analysis about consistency, validation and evidence. Big, strategic decisions need to be based on quantifiable facts, not generalizations. At Thematic, we consider ChatGPT as a tool, but not a solution that gives you the level of accuracy and insight you need.

  • Accurate and fast insights with LLMs and our own algorithms combined

To deliver specific and accurate answers, we use large language models (LLMs) such as GPT4 along with our own algorithms. This makes it easier and faster to get granular and reliable answers.  That’s because our proprietary algorithms do the work to ensure that generative AI only analyzes useful and clean data.

Part of this process - a really important part - is that a human validates Thematic's AI's analysis. With our built-in tools to guide the AI, a human can check the predictions, and refine themes to fit their company's decision needs. This makes it easier to get specific insights for decision-making and capture the full voice of the customer. It dramatically reduces the time it takes to produce reports relating to a specific inquiry.

  • Analyze large volumes of feedback

A significant accuracy advantage comes from Thematic’s capacity to process larger amounts of feedback, without needing to do batch analysis. Thematic can generate all the relevant themes (up to 2000, if you wish) while ChatGPTs has a very limited scope.

Thematic summarized the issues with Twitter, from over 6000 app reviews

If GPT did analyze all the unstructured feedback data at once, it would get all the noise and irrelevant comments, resulting in irrelevant themes. If you ask GPT to rerun the analysis, results will be inconsistent, because it will draw on a different batch of feedback.

  • Reliable data security & compliance

Importantly, you know that your data is secure. Thematic uses OpenAI through Azure, meaning it is GDPR and SOC2 Type II compliant. With GPT, there are risks around the security of your data and compliance with EU regulations and data is potentially used for training purposes. If your organization cares about privacy and protection of data (and it should), OpenAI offers you no guarantee about where your data is consequently stored or used.

  • Visualization for insights & decision making

Thematic’s user interface is built specifically to help decision makers get all the layers of data insights they need to deliver to their stakeholders, by identifying and visualizing the drivers of satisfaction or score changes. Emerging trends in customer feedback and their impact on the business can easily be gleaned through reporting. We've identified what insights leaders need to deliver to their stakeholders and ensured that they get all that in one platform.

Dive deeper into Thematic with LLMs

Thematic CEO and Co-founder, Alyona, recently shared her experience experimenting with ChatGPT to analyze feedback - read about her experience, how it worked and where it fell short.

If you’d like to explore the insights you can get with Thematic with generative AI, book some time with our helpful team to get a free demo on your data.

Ready to scale customer insights from feedback?

Our experts will show you how Thematic works, how to discover pain points and track the ROI of decisions. To access your free trial, book a personal demo today.

Recent posts

How to theme qualitative data using thematic analysis software
How to theme qualitative data using thematic analysis software
Members Public

Become a qualitative theming pro! Creating a perfect code frame is hard, but thematic analysis software makes the process much easier.

AI & NLP
How to super-charge your Qualtrics setup with Thematic & Power BI
How to super-charge your Qualtrics setup with Thematic & Power BI
Members Public

Qualtrics is one of the most well-known and powerful Customer Feedback Management platforms. But even so, it has limitations. We recently hosted a live panel where data analysts from two well-known brands shared their experiences with Qualtrics, and how they extended this platform’s capabilities. Below, we’ll share the

Customer Experience
How we use our own platform and Chrome extension to centralize & analyze feedback
How we use our own platform and Chrome extension to centralize & analyze feedback
Members Public

Customer feedback doesn't have all the answers. But it has critical insights for strategy and prioritization. Thematic is a B2B SaaS company. We aren't swimming in feedback. Every piece of feedback counts. Collecting and analyzing this feedback requires a different approach. We receive feedback from many places: * our in-product NPS

Feedback Analysis