Ensuring AI Transparency and Trust: How Thematic aligns with ESOMAR's Guidelines for AI buyers
AI has many benefits for market researchers and insights analysts. It reduces time-consuming tasks, delivers deeper insights faster, and unlocks time for strategic work. However, AI also comes with risks that can be challenging to assess and mitigate, especially in the crowded AI marketplace. Knowing what's AI and what's AI washing is difficult and there are limited guidelines for evaluating AI solutions.
To assist buyers in navigating these challenges, ESOMAR, an independent global hub for research insights and analytics, created a list of 20 questions. These questions help evaluate an AI provider’s reputation, capabilities, human oversight, and data governance protocols. In this post, we will discuss how we solve for these important guidelines at Thematic.
Before reviewing our answers, make sure to download the full ESOMAR guide for AI buyers that provides context for each question and what to look out for.
A. Thematic company profile
This section will help you understand our credentials.
1. What experience and know-how does Thematic have in providing AI-based solutions for research?
Thematic was founded by Alyona Medelyan, PhD. She started her journey in AI in 1998. Her Masters and PhD (sponsored by Google) were specifically in the area of making sense of text using algorithms. Her academic research resulted in algorithms used by hundreds of organizations. Since graduating, Dr. Medelyan worked in commercializing AI technology to help startups build their first products and established companies add AI to their offering. She founded Thematic, when several companies asked her help them understand what's driving their NPS.
By speaking with market research experts, Dr. Medelyan quickly learned the importance of transparency and trust over a black-box AI. Thematic's core solution was built around the idea of human and AI collaboration.
Our R&D team at Thematic is dedicated to continuously evolving our proprietary Hybrid AI, integrating both in-house and third-party algorithms and models. We believe that Large Language Models have solved nearly all tricky language understanding problems. But there are trade-offs that come from using them. With over 20 years of experience in AI commercialization, we specialize in expertly managing these trade-offs to deliver optimal solutions.
2. Where can AI-based services have a positive impact for research? What features and benefits does AI bring, and what problems does it address?
At Thematic, our mission is to get insights from every voice into the hands of every decision maker. This task typically falls into the hands of researchers working in Voice of Customer, Customer Experience and Customer Insights teams. These teams tend to be small, under-resourced and overwhelmed with requests. They often struggle with analyzing qualitative feedback, which is siloed, scattered across tools, and noisy.
There are 3 ways in which AI helps researchers solve these problems and delivers benefits:
- Faster data analysis. Between real-time transcription, AI-generated summaries, and accurate theme identification, AI saves researchers hours (and hours!) of manual work.
- Work at scale. UX and VOC teams can now collect and analyze huge amounts of data, which means they can source feedback from a larger number of participants.
- Deeper, more informed insights. AI can spot patterns and read between the lines, providing researchers with insights that directly impact business decision-making.
3. What are the practical problems and issues in the use and deployment of AI? What has worked well and how, and what has worked less well and why?
Some people say that a Large Language Model is an oddly shaped tool. There has never been a tool that lets you ask any question and return a reasonable answer. The answer quality typically depends on the quality of the question you ask, and the quality of the model.
With that, there are several key problems that we had to solve when integrating LLMs in Thematic:
As you can see, there is no quick solution. What worked well for us is:
A) Narrow down individual tasks, for example, find broad themes, find sentiment of a sentence, auto-score customer conversation.
B) Make sure we provide high-quality specific context for that task, that reduces hallucinations and increases consistency.
What doesn't work well is keeping the task broad. For example, if you were to ask a Large Language Model "here's my dataset, what are the insights?", you will get a different set of insights every time.
B. Is Thematic's AI capabilities explainable and fit for purpose?
This section will help you evaluate our services from a practical standpoint and whether our capabilities align with your business purpose and are likely to provide clear benefits.
4. What's the role of AI in Thematic? What are the key functionalities? (Explained in simple, non-technical terms)
Thematic is a best-of-breed AI solution for analysis of unstructured customer feedback. So, AI plays a major part of our offering. There wouldn't be Thematic without AI.
Here's a list of our key AI functionalities:
- Thematic Analysis - Thematic uses AI to discover themes in text feedback and organize them into a custom taxonomy. Themes emerge organically from what customers are saying, so you can discover the unknown unknowns, and get as granular as possible. Using AI allows you to analyze a lot of data quickly, while still being accurate.
- Themes Editor - One of Thematic's incredible points of difference is our themes editor. Once the AI has completed its analysis, you can use our no-code tool to merge, split or rename themes, discover new themes, and add your unique understanding of your product and company needs. This ensures your themes are managed and structured in a way that is relevant, accurate and specific to how your organization acts on feedback.
- Sentiment Analysis - Thematic assigns a granular sentiment score to each segment of text and the corresponding theme. Tagging feedback with a sentiment score is especially helpful when feedback doesn't have a rating or score such as NPS.
- Category Analysis - Thematic discovers sentence segments that express issues or bugs, feature requests and questions in feedback. This is helpful when pulling out the specific aspects of customer needs from feedback.
- Synthetic Scores - Thematic assigns custom scores and ratings, such as CSAT, Customer Effort Score, NPS, likelihood to churn, or whether a call center call was resolved or unresolved, and so on. This helps identify themes linked to undesirable customer outcomes.
- Summarizer - Another timesaver is our theme summary feature. Our AI algorithm pulls out sentences on a specific theme and groups them by meaning into clusters. This gives you an instant deep-dive into a theme without having to read through thousands of comments. We also use summaries to get an overview of the dataset or compare two different data segments.
- Thematic Answers - Users can ask questions in natural language about customer feedback, and do it across datasets. For example, what do people like, how did we do this quarter vs. last quarter, what's driving our NPS and many more. Thematic interprets the question, pulls relevant data and automatically generates a report, with ability to deep-dive or create suggestions on how to act on this feedback.
- Conversational Analytics - Thematic analyzes chats and call transcripts to pull out root cause and summarize both user needs and agent's actions. Each sentence within this summary can be easily traced back to the source for verification.
5. What is the AI model used in Thematic? Are our AI solutions primarily developed internally or do they integrate an existing AI system/ third party, and if so, which?
Thematic develops all AI solutions internally. We have a dedicated and well-oiled R&D team that runs rigorous process to build, evaluate and launch AI features in production. We use a variety of AI models, some of which are developed internally, and others delivered by third parties. We most commonly use with AWS Bedrock and Microsoft Azure Cognitive Services, which lets us use Large Language Models in a secure way.
Please note that we are are constantly updating our underlying AI models, as new models are released. We use an evaluation framework that is constantly evaluating model performance on individual tasks and picks the model that provides the best trade-off of speed, accuracy and cost.
6. How do the algorithms deployed deliver the desired results? What's the underlying data and the way in which it interacts with the model to train your AI service?
Our proprietary non-generative AI models have been trained on a large volume of customer feedback that we gathered over the past 10 years with explicit permission of our customers. Before training, all data was de-identified to protect its origins. All customers had the option to opt-out from this process.
With the arrival of Large Language Models, we have stopped training custom models. We never train LLMs in-house or share our data with LLM providers for training purposes. Instead, we deliver accuracy through prompt optimization and a complex evaluation process involving small amounts of manually labeled data.
C. Is Thematic's AI capability trustworthy, ethical and transparent?
This section will clarify our ethical principles and our view on important topics such as biases, data security and resilience.
7. How does Thematic ensure the accuracy, validity, and reliability of its AI-generated insights, and what processes are in place to handle unreliable or biased outputs? How do we ensure that the results generated are ‘fit for purpose’?
Accuracy of the output is top of mind for us at Thematic. We offer many ways to verify and validate the accuracy of its themes, sentiment, answer and summary quality. We also offer automated quality assurance with recommended actions to take to improve unreliable results. The main idea is that AI makes suggestions and user accepts or rejects them, while seeing the impact these changes have on quality.
8. What are the limitations of Thematic AI models and how do we mitigate them?
While our AI models are powerful tools for feedback analysis, like all technology, they have their limitations. They may not fully capture nuanced human emotions or context-specific language depending on the dataset. To mitigate this, we continuously update our models and prompts. We also leverage human oversight to interpret complex insights, ensuring the highest accuracy and relevance in our analyses. This blend of AI precision and human expertise helps us deliver balanced and insightful feedback solutions
9. What considerations, if any, have you taken into account, to design your service with a duty of care to humans in mind?
At Thematic, we take our duty of care seriously when designing our AI solution. We have outlined and are following the specific guidelines we deploy for the use of Generative AI at Thematic. We see ourselves as the guardians of your data, making sure it's anonymized and handled with the utmost security. We're also committed to fairness, using smart techniques to spot and tackle bias, so you get insights you can truly trust. Our goal is to keep everything transparent and empower you with the knowledge to make confident decisions, making sure our technology lifts everyone up in the process.
D. How do you provide Human Oversight of your AI system?
This section will explain human involvement and oversight, both in development and use of Thematic. You'll learn what role the human plays when working with Thematic's AI.
10. Transparency: How do we ensure that it is clear when AI is being used in any part of the service?
The vast majority of Thematic is AI powered, so by default all analytics are AI generated. The model created is completely transparent and the user can introspect it and the results at any time.
We believe it is necessary to understand and evaluate how AI has analyzed data, to ensure the system delivers a high quality of information and supports trust in the insights.
Where generative AI is used we make it obvious and we keep track of user review cycles, so that it's clear which analysis was validated by a human.
11. What are the explicit ethical principles behind Thematic? And how do these principles help determine AI's behavior? How do we ensure that human- defined ethical principles are the governing force behind Thematic?
Thematic is guided by clear ethical principles, focusing on fairness, privacy, and transparency. These principles shape how our AI behaves, ensuring it treats all data equally, protects your privacy, and operates in a clear and understandable way. We ensure analysts can see exactly why decisions are being made by the model and provide tools for them to determine if bias has been introduced. Our products are developed by a diverse teams who spot and address biases at all stages of development. We also maintain strict data privacy standards. By keeping these values at the core, we ensure our AI serves and respects everyone involved.
12. Responsible Innovation: How does Thematic integrate human oversight to ensure ethical compliance?
At Thematic, we integrate human oversight by having dedicated teams review and audit our AI models, ensuring they align with ethical standards. This includes both staff within Thematic, and at our customers. This involves continual verification of data accuracy, and assessing the impact of AI-driven insights. We involve cross-functional teams in decision-making processes to maintain diverse perspectives.
E. What are Thematic's Data Governance protocols?
In this section, we'll help you understand our approach to data protection, information security, compliance policies, procedures and practices.
13. Data quality: How do we assess that the training data used for AI models is accurate, complete, and relevant to the research objectives in the interests of reliable results and as required by some data privacy laws?
At Thematic, we rely solely on Microsoft and AWS for our Generative AI models. We trust their top-tier data practices to ensure our training data is accurate, complete, and relevant. Their strong commitment to data integrity and privacy aligns perfectly with our goals and legal requirements. By partnering with them, we ensure our AI provides trustworthy insights while upholding data security and confidentiality.
Our customer's data is only ever used to train models wholly within that customer's workspace and control. Data can not be used for any other purpose.
14. Data lineage: Do we document the origin and processing of training or input data, and are these sources made available?
At Thematic, we meticulously document the origin and processing of training and input data in our internal knowledge base. This is critical as transparency and ownership are key value of our company. This information is available to authorized stakeholders to ensure data integrity and trust.
Models controlled by customers also have full documentation of how and when they were created along with any modifications by the customer.
15. What's our privacy notice?
Here's the link to our privacy policy.
16. What steps do we take to comply with data protection laws?
To comply with data protection laws, Thematic implements strict data anonymization and encryption protocols, ensures transparent consent processes. Every year we undergo a strict audit under our SOC2 TypeII compliance, to make sure that we comply with all legal data handling practices, and keep abreast of the latest legal requirements to maintain robust privacy measures.
We are compliant with the GDPR, CCPA and other legislative requirements.
Please note, that the full ESOMAR question talks about protection of research participants. In Thematic, we do not work with research participants. We only analyze public data or data that the company already gathered from their customers, prospects, or employees.
17. What steps do we follow to ensure Thematic is resilient to adversarial attacks, noise and other potential disruptions? Which information security frameworks and standards do we use?
To enhance Thematic's resilience against potential threats, we conduct regular penetration testing to identify and address vulnerabilities in our systems proactively. Additionally, we comply with the SOC 2 Type II standard, which involves comprehensive audits of our security controls to ensure the confidentiality, integrity, and availability of our data. These measures collectively reinforce our commitment to maintaining a secure and reliable platform for our clients.
Security, availability and confidentiality are considered at all stages of the software development lifecycle. Where possible we use the NIST frameworks to match industry best practices.
18. Data ownership: Do you clearly define and communicate the ownership of data, including intellectual property rights and usage permissions?
At Thematic, we clearly define and communicate data ownership, intellectual property rights, and usage permissions to all stakeholders. We ensure that our clients understand how their data is used and stored.
In the video below, see how we prioritize data and insights governance within the Thematic platform.
19. Data sovereignty: Do you restrict what can be done with the data?
Our system prioritizes reliability, privacy, and data sovereignty. We guarantee that AI learnings remain the property of the customer. We uphold stringent data sovereignty and privacy standards, ensuring that data remains within the specified region or data center. By leveraging AWS's robust security and privacy measures, we maintain data protection and compliance, allowing us to concentrate on delivering our core services responsibly.
Each customer is assigned a particular geographic region and no services outside that region are used in the delivery of Thematic's services.
20. Ownership: Are you clear about who owns the output?
At Thematic, we clearly establish that the ownership of the output generated by our AI models resides with the customer. We ensure this understanding is part of our agreements, providing full transparency and control over the insights and results produced from their data.
Conclusion
In this post, Thematic responds to important questions from ESOMAR to clearly and transparently share how we use AI. We explains that AI is used throughout Thematic, to integrate, analyze and report on feedback data. We pride ourselves on holding ourselves to the highest standards of integrity, ethical principles and transparency with data and AI.
Book a demo today to explore how our AI data analytics will make it faster and easier to get your Voice of Customer, CX and product insights.