What’s the best way to analyze vast amounts of feedback?

Do you have a lot of customer feedback collected but don’t know exactly what to do with it?

Maybe you’re debating whether to hire a data scientist in-house to analyze it all manually, or go the agency route? Perhaps you’ve heard about text analytics and wonder what it’s all about, and what you need in-house to get started.

Here, we go through the pros and cons of each of these solutions.

In this blog, I’ll aim to answer these questions:

Why do you need a customer feedback analysis solution in the first place? Is conducting surveys not enough?
Do you hire 1) someone internally to analyze your data manually or using data science techniques, 2) hire a third party agency or 3) get an online software service (SaaS)?

1. Why do you need a feedback analysis solution in the first place? Is conducting surveys not enough?

Today, many companies are, what I call, “data rich but insights poor”. You may have lots of feedback collected; heck, you may be swimming in customer feedback, but does it really give you the data-driven insights you’re after?

Oftentimes, companies with the best intentions find themselves in an insights vacuum. You know you need insights to inform your decision making and you know that you’re lacking them, but don’t know how best to get them.

Conducting surveys is simply not enough. Consider this example. You run a restaurant and you conduct a survey asking respondents to rate your customer service on a scale of 0 to 10.

Like most people, you spend the bulk of your time running your business, your “day job”, and executing the survey and collating the results. So, often actually analyzing the customer feedback results becomes an afterthought. Maybe you even calculate the score across different segments. But does the score really identify the information you need to improve your business? Not likely.

“Scoreboard watching” is a trap even the best organizations fall into. But the score itself is not truly insights. Scores only matter when combined with customer feedback that’s meaningful, e.g. answers to open-ended questions or customer interviews.

So, you’ve just spent lots of time and resource collecting data, and maybe you’re making assumptions about the results by reading some of it.

How can you know for sure what your customers are in need of, if you don’t scientifically analyze it? What are you missing if you don’t analyze your data properly? You also need to make sure your analysis of that customer feedback is actionable to inform your growth strategies. Otherwise, what’s the point?

4 benefits of effective feedback analysis

We recently published a great list of why you need to properly analyze your customer feedback, and it includes things such as:

1. More business growth

By analyzing feedback and in turn, listening to your customers, you can increase upselling and cross-selling success rates by 15% to 20% (Gartner).
This means improved customer retention, less churn, and a higher customer lifetime value, plus less costs to retain buyers, (Gartner).

2. Improve your Net Promoter Score

By truly listening to your customers and addressing their biggest concerns, you should expect your Net Promoter Score to be positively affected.

3. Better customer experience

Listening to your customers ultimately means creating a better customer experience over time by driving positive change in the right places.

You’ll know where it hurts, know where to improve and how to improve it; and because you’ll analyze your feedback in the right way (more on this later) – you’ll get these insights fast.

4. Better products and services

Analyzing customer feedback to discover what matters to customers - whether that's bugs to fix, new features to build, or user issues getting in the way - you will help your product managers and engineers develop better products that your customers will appreciate more, as they’ll better fit their requirements.

How to analyze customer feedback

There are 4 types of solutions for analyzing customer feedback:

Outsourced data analyst
Hire a data analyst
Outsource agency
SaaS products

For the remainder of this article, we’ll discuss the pros and cons of each.

2. Do you hire someone internally to analyze, hire an agency or get online software?

This is the most common question we hear from our customers and partners. Without a clear understanding of the pros and cons, companies end up delaying the analysis and therefore miss out on all of the benefits (faster growth, better customer retention). So, here is a comprehensive list of pros and cons of each approach.

A. Task a data analyst (or hire one) to conduct manual data analysis – pros and cons

Pros:

Insider knowledge. Data analysts, who may have been with you for a few years, typically know your company through and through, and have any type of insider knowledge needed to analyze customer feedback. Any new people you hire new for this role would need to be trained up first.
Common sense. People have an incredible depth of common sense required to separate actionable from non-actionable feedback. However, tread with caution here too: this comes with a cost of bias (see cons).
Language knowledge. With all of the advancements in AI, truly understanding natural language and its nuances is still a skill best mastered by people. This is especially true for sarcasm. For example, if somebody writes “Great service, yeah right!” and the algorithm tags it as positive.

Cons:

The cost to find and retain the right candidate. Hiring new staff means time and resource invested by your company. Typically, people underestimate the cost of hiring, which includes time to create a job ad with the right description, publishing it, sourcing and interviewing candidates, wait time for the candidate to accept the offer and be available to start, training them on the ins and outs of the organization and so on.
Human bias. If you have someone on staff who can do the job, can you 100% rely on them doing this job accurately and efficiently? I spoke with an analyst today who said “I love reading customer comments, however, it’s a time sink. I end up reading each comment five times to make sure I’m not reading into it my own interpretation. I want to make sure our management gets an accurate view, and it takes time to verify it.” Cognitive bias is a systematic error in thinking, that affects the decisions and judgments that people make, which can be related to memory. The way a person remembers an event may be biased and that can lead to biased thinking and decision-making. So, it’s virtually impossible that some form of bias does not impact the results even by the most intelligent people.
Human error. Have you ever tried to summarize the opinions of 100 people by hand? It’s really hard. Most people get distracted at such repetitive tasks, and ultimately miss things that lead to errors.
Time to action. Data analysts typically spend days if not weeks preparing their reports. This is the most time-consuming way to analyze feedback.
Knowledge sharing. Another downside is that the full responsibility lies with one person, who sits on all of this knowledge. What happens is anything were to happen to this person, how do you back yourself up?

If you do choose to do your analysis in-house, here’s a great piece on how to manually analyze open-ended questions. And here’s a great example of how to best get actionable insights from your customer feedback.

B. Hire a data scientist to automate data analysis – pros and cons

Pros:

Scalability: Data analysts can only analyze a limited amount of feedback manually. A data scientist can put together a solution that can scale to millions of feedback pieces, and “the only” costs to the company would be infrastructure (either internal on-premise, or external, e.g. an AWS, Google Cloud or an Azure server) and the salary.
Insider knowledge. Data analysts have access to insider knowledge needed to make decisions about how to analyze customer feedback. If they don’t have the answers, they can more easily find time to speak with the right people in the company than an external party.
Data security. A data scientist can design a solution that is run on-premise, meaning that none of the customer data leaves the enterprise. It does not mean that this is the most secure solution, as it will only be as secure as the data privacy training received by the data scientist.

Cons:

Availability of the right talent. We’ve already discussed the costs to find and retain the right talent above, under A (hiring a data analyst). When it comes to data scientists, expect an even tougher time to find the right person. Let’s examine the kind of person you need ideally:
- Since the most precious type of customer feedback is derived from natural language and it’s hard to analyze, you would need a data scientist skilled in Natural Language Processing. These people need a great knowledge of linguistics and programming, something that isn’t covered by a mainstream data science education.
- If you can’t find such a data scientist, consider outsourcing the thematic analysis to a vendor, but still use a data scientist to combine it with internal financial data, such as how many times a customer has contacted you in the past month, or what the exact items are that the customer bought, to drive deeper insights.
- You will also need someone who has the experience or the smarts to understand the actual problem you are trying to solve and how businesses work.

Customer feedback analysis should be a process, not a project. Data scientists are great at running short-term experiments to find answers to specific questions, but in order to turn customer feedback analysis into a company-wide process, you will need to create an end-to-end system that ingests feedback, analyzes it and lets you explore it on a weekly or monthly basis. In order to create such a solution, the data scientist would need to work alongside a software engineer, and ideally a UX designer, to create this solution. Yes, it’s the size of a small startup. They could create a custom solution for your company, but it will cost a lot of money to source these people, free them up from other projects, create this solution and, ultimately, maintain the solution over time, as participants may switch jobs.
Distraction from the main company focus. Are you in the business of collecting or analyzing customer feedback? If not, then chances are that dedicating precious data scientist time to this task is not the best use of their time. Today, companies need to focus on providing exceptional experiences to their customers. Customers won’t value the time you spent on creating internal tools, but they will value new products and features of the existing products that serve them better. For example, a data scientist who works in a software company could be working on sourcing data that improves the actual software that the company builds. A data scientist who works at an airline could be using data to optimize the time of airfare promotions. Think about what a data scientist could do to help your core business.

C. Hire an agency to analyze feedback – pros and cons

By this I mean, completely outsourcing all of your customer feedback analysis to a third party. Traditionally, companies would employ market research agencies and only get the results in the form of Powerpoint reports. Today, some agencies provide dashboards built-in tools like Tableau and Power BI. Others, help you decide which vendor to source for your project.

Please note, the question of whether to manually analyze data or take advantage of data scientist, or text analytics vendors may still prevail, as agencies will have different ways of analyzing the data. For the purpose of this review, we’ll just focus on the pros and cons of involving an agency, regardless of their approach.

Pros.

Access to expertise. Great agencies employ people skilled in market research. They can advise you on how to structure your customer surveys, on how to focus on the right questions to ask, and those who helped other companies with similar projects will have the benefit of experiences others had in the past. So you can benefit from other people’s mistakes and learnings, and won’t need to hire expensive staff members yourself.
Unbiased competitive analysis. Another great benefit of using agencies is that they can source feedback from your competitors’ customers. Nothing is more powerful than seeing how you are perceived in relation to other companies, and it’s a huge bonus to be able to benchmark yourself.

Cons.

Costs. Outsourcing can be excruciatingly expensive. If budget is no issue for you then this may not be a problem, but if it is, get ready to tighten those purse strings.
Delays. Communicating with an external party ultimately results in delays in getting the insights and actioning on them.
Lack of internal knowledge. If you choose this type of solution, a lot is riding on the relationship you have with your agency. You might be able to ask them to look into certain factors in your data and deliver you reports, but in most cases, the power lies with them. You likely will not have the opportunity to dig deeper and look into themes and trends by yourself, in your own time, but you’ll have to ask them every time you do and liaise with an account manager (or – gasp – a junior account manager!).
One time snapshot report. Often agencies product a one-off report that you spend a lot of money on and can’t easily track progress without commissioning the same amount of work again.

D. Software as a service (SaaS) – pros and cons

SaaS has revolutionized the world of business. For every single business process, there is now a solution you can buy, built by people who care deeply about the specific problem that needs solving. And in most cases, it’s more cost-effective to buy than build your own.

There are three types of SaaS products that can help you with customer feedback analysis:

1. Customer feedback management solutions

These range from specific collectors of NPS feedback like Delighted to huge platforms like Medallia that not only collect, but also make it easy to respond to each comment and keep track of your scores. These solutions typically focus on these specific workflows, and not the analysis of the data. As a result, the analysis ends up being either too basic (word clouds) or too complex (require professional services to even set things up, or another full-time staff to make sense of the results).

2. Text analytics tools and APIs

These are typically tools for data scientist or analysts to load the text and analyze it inside these solutions. Some examples are companies like Lexalytics or Gavagai that are designed to tackle just any text, and not customer feedback specifically. Same happens with Google NLP and similar APIs, which tend to work well on tasks such as finding names of locations in text, but fail when it comes to merging customer feedback on the same topic.

3. Customer insights solutions

These are new SaaS offerings that focus on getting actionable insights from customer data, including customer feedback. At the core, these solutions use text analytics as well, but their algorithms are tailored to work on short customer feedback text.

The positives of using customer insights solutions far outnumber the negatives (and yes, we may be biased!), but we’ll keep to the biggest pros here.

Firstly, let’s clarify: what is text analytics that’s at the core of solutions like Thematic. Text analytics is the process of using machine learning and natural language processing (NLP) to extract meaning out of text. For example, this can be analyzing text written by customers in a customer survey, with the focus on finding common themes and trends. To make text analytics the most efficient, organisations can use text analytics software, leveraging machine learning and natural language processing algorithms to find meaning in enormous amounts of text.

Here at Thematic, we use text analytics to analyze the free-text feedback submitted in customer feedback forms. Thus, we help companies find hidden customer insights and find recurrent and emerging themes, tracking trends and issues, and create visual reports for managers to track whether they are closing the loop with the end customer.

Pros.

Resource-light. Imagine that you can have someone who is simply designated to be responsible for improving your customer experience and receiving the (already analyzed) customer feedback reports, but does not need the data science degree to be able to make sense of the insights..
Fast. If you have the right solution at hand, you can be set up in a matter of days, if not a couple of weeks, and then instantly have all data analyzed and turned into reports.
Unbiased. Text analytics is also more reliable than having a person manually analyze feedback. Falling back on a systematic AI approach, there’s no room for human bias or in fact, error.
Easy access and sharing with your team. Depending on your solution, if you have an online dashboard, which many of your team members can access, you can also dip into your data anytime and analyze what you want. At Thematic, our clients use our online dashboard to check recurring themes, the impact on their NPS score, compare with their main competitors, check themes over time, and easily customize data.
Low maintenance. Software is always updated with new features, you don’t need to maintain it. Vendors train up new staff when required, as they are motivated by having you use the tools. The value you are getting is constantly increasing too, because the technology gets better and better each time.

Cons:

Data concerns. Customer feedback is a precious resource, and so ideally, you need to rely on external vendor’s security policies to make sure the data stays save once you share it outside. We recommend two approaches here: a) Look out for internationally recognized certifications, such as SOC2 and ISO-27001. b) Only share feedback data, but never names, email addresses and other personal data.
Ability to bring in domain expertise. Some solutions can be limited in tailoring the results to your specific needs. Look out for vendors who make it easy to bring in your internal company knowledge to make sure that the results are relevant to your business.
Costs. Some solutions can be expensive and not scalable. Weigh in the pros and cons of building your own (see point B) and get quotes early.

They can quickly see the relative importance of each theme and subtheme and compare the differences between two segments of data. They can calculate not just what is mentioned the most, but also what each theme’s impact is on overall customer satisfaction and loyalty, and easily understand what drives the scores. The themes are also easily adjusted to suit their company terminology and unique needs using our simple drag and drop UI.

Want to learn more about the different types of text analytics? Here’s a great resource we created which includes different types of analysis and their pros and cons.

Wondering what are the pros and cons of a manual vs analytics solution? Here is an e-book that compares manual analysis of NPS feedback with text analytics. Which do you think works best?

Here’s a bit more info on how we do it and here, we’ll show you how to quickly find insights in free-text feedback, using Thematic.