With ever more AI tools available on the market, organizations need better ways to evaluate how good they are at their job. Cassie Kozyrkov, Google's Former Chief Decision Scientist, has recently built a set of criteria to assess the quality of AI decision-making. We decided to apply her “Kozyr criteria” to Thematic’s AI-powered feedback analytics tool to find out whether it would pass the test.

Why It’s Important to Understand How AI Analyzes Your Data

The recent McKinsey Global Survey reported that 72% of organizations have now adopted AI for at least one business function. And 65% are already using Generative AI. Yet at the same time, 63% of these businesses are worried about inaccuracies with 38% actively working to minimize this risk.

Understanding exactly how AI is analyzing your data is key if you want greater accuracy and better data-driven decisions. Traditional AI systems are straightforward and seemingly easy to understand. But they are usually created with a big set of rules. While you can see how the data was analyzed, it's challenging to look at the rules and keep the rules up to date. It can also be costly to get insights customized to your business.

On the other end of the spectrum, many modern AI solutions like GPT closely guard their processes. The AI most companies use today is basically a black box. This makes it hard for users to understand exactly what is going on inside.

Here at Thematic, we give users greater transparency so they can understand how the AI has analyzed their data. This means you can go in and tweak things during the analysis process. For example, ​​you can adjust and personalize the “themes” or topics the AI detects in your data to ensure better alignment with specific project requirements.

Giving users greater control and oversight helps ensure a higher quality of information. And this in turn supports business leaders to trust the insights and make better decisions.

What are Cassie Kozyrkov’s 3 criteria?

Cassie Kozyrkov founded the field of Decision Intelligence while working as Google’s first Chief Decision Scientist. In simple terms, Decision Intelligence breaks down the science of making decisions. And as AI usage continues to gather pace globally, Kozyrkov is particularly interested in studying "the decision-maker behind the curtain".

Kozyrkov recently shared three key criteria she uses to evaluate AI systems. These are known as the Kozyr criteria:

1. Identify the AI system’s objective/s: Identify what the AI system is built to do.
2. Assess data access: Check if AI has access to the right data for the job it's meant to do.
3. Functionality: The final stage is to identify if AI works as intended.

"If you know these three things, you know a lot about the system," Kozyrkov says. "You know a lot about its risks and how it might benefit some people more than others, for example. And you know a lot about what could go wrong with it right off the bat."

How Thematic Uses the Kozyr Criteria to Validate its AI System

Whenever Thematic's AI is deployed on a new project, we evaluate how our AI system will support effective decision-making for the use case. We decided to apply the “Kozyr criteria” to Thematic’s AI-powered feedback analytics tool to find out whether it would pass the test.

1. What is the objective of Thematic?

Thematic's AI-powered platform was designed to transform large volumes of unstructured data into a clear understanding of issues. These insights give organizations the information they need to make better decisions and drive business growth.

For example, a company like Albertsons might want to track customer reactions to a new delivery system. By collating and analyzing customer reviews and social media comments in real-time they could track how their customers respond to the system. They can fine their messaging to address issues or even change the system if their customers really don’t like it!

2. What data will the AI use, and is it the right data for the job?

Thematic works with a huge variety of unstructured data. This might include survey responses, product reviews, or customer support interactions. Thematic's extensive data-cleaning library verifies that the data does not have duplicates and checks that all sensitive information is redacted and the data is complete and current.

There is also an interface in the platform where users can validate the quality of the source data. This builds trust and confidence that your insights are generated on relevant data.

Thematic was built after testing on thousands of millions of feedback pieces across a range of industries. The AI is built to understand the idiosyncrasies of feedback. For example, customers often use slang and emojis, make spelling mistakes, or even write nonsensical text. Thematic automatically cleans and prepares your data so it can be analyzed effectively.

3. Evaluating Your Insights for Accuracy & Consistency

To make good decisions you need accurate and consistent information. A human may deliver a different answer on a different day, depending on what else has influenced them. Generative AI tools will often be inconsistent as the AI draws from a different batch of data each time.

Thematic has a number of useful features that help you evaluate and refine your insights as you go:

Tools for Double-Checking Your Analysis

If you type a question into Thematic Answers, you can check that the AI delivers the narrative answer and quantified themes when drawing from the same data.

You can also double-check your analysis by comparing it with manual or pre-existing analysis. For example, you could run 200 rows of data through Thematic and see how it compares to your existing analysis.

Throughout Thematic's platform you can easily see how the AI has analyzed a phrase. Themes and sentiment are highlighted in the original feedback data, and its metadata can be verified in one click. Users can easily review if the AI delivers results that make sense.

Let’s take the example of a piece of feedback on a webcam, "Gets the job done, but it's not cheap!". In Thematic you can quickly glance at the original comment to see if both the positive and the negative sentiment are highlighted, along with themes of “Product Functions”, “Price”, and “Value” for money.

Human-in-the-Loop: Assessing and Refining Your Results

Thematic goes one step further than just AI analytics. The platform combines the best of human and AI by enabling analysts to refine their results manually. Thematic’s Themes Editor allows human users to access and make edits to the themes and codes created by our AI.

For example, you may decide that certain themes are not relevant and remove them from your insights. Or you may want to edit themes for greater clarity or alignment. The AI then learns from your refinements and applies them to your data at scale.

Conclusion: Transparent AI Means Better Decision-Making

Thematic is specifically designed for decision makers to get deep layers of insights from feedback data. Even non-technical stakeholders can see how the AI has analyzed the data using the platform’s clear visualizations. And analysts can take it a step further by adding a human touch, refining how themes are structured and analyzed in real-time.

Thanks to these unique features Thematic's AI meets decision makers’ needs, and it passes the Kozyr test!

Try Out Thematic On Your Data

AI-powered tools are only as good as the data that goes into them. And that’s why we encourage businesses to try out Thematic on their own data with a free guided trial.