Complete Guide to Sentiment Analysis: Updated 2020
What is sentiment analysis? If we take your customer feedback as an example, sentiment analysis (a form of text analytics) measures the attitude of the customer towards the aspects of a service or product which they describe in text.
This typically involves taking a piece of text, whether it’s a sentence, a comment or an entire document and returning a “score” that measures how positive or negative the text is.
For example in customer feedback:
We’ve also heard sentiment analysis being referred to (less commonly) as opinion mining and emotion AI.
Why sentiment analysis is needed
In today’s environment where we’re suffering from data overload (although this does not mean better or deeper insights), companies might have mountains of customer feedback collected. Yet for mere humans, it’s still impossible to analyze it manually without any sort of error or bias.
Oftentimes, companies with the best intentions find themselves in an insights vacuum. You know you need insights to inform your decision making. And you know that you’re lacking them. But you don’t know how best to get them.
Sentiment analysis provides answers into what the most important issues are. Because sentiment analysis can be automated, decisions can be made based on a significant amount of data rather than plain intuition that isn’t always right.
Impossible to analyze large amounts of data without error
Imagine this scenario: you’re the owner of a small delivery business and you receive about 20 responses to your email surveys every month. You could (and should), read these yourself and perform your own analysis by hand.
Now, imagine receiving 30,000 responses per month. That’s more than a thousand responses each day! Needless to say this is impossible as a part of a business owner’s day job.
Then, there’s the question of bias. Everyone knows “those days” where things go wrong and you’re in a foul mood even before reaching the office. The risk of you interpreting messages and any form of communication more negatively, is rife.
You might also have your own, preconceived opinions about the topic at hand. All of this can influence how you interpret the text you need to analyze.
You’ll also need to summarize the feedback into a few actionable insights, so that it is meaningful for your company to make use of.
Also the insights need to be translated into presentable form so that it is easy to grasp.
Sentiment analysis is important because companies want their brand being perceived positively, or at least more positively than the brands of competitors.
What sentiment analysis is used for
Sentiment analysis is useful for quickly gaining insights using large volumes of text data. In addition to the customer feedback analysis use case here are another two exemplary use cases:
One example is stock trading companies who trawl the internet for news. Here, sentiment algorithms can detect particular companies who show a positive sentiment in news articles. This can mean a significant financial opportunity, as this may trigger people to buy more of the company’s stock. Having access to this type of data gives traders the opportunity to make decisions before the market reacts.
Below is an example of how stock price of a company can be affected by news. The sentiment expressed in the news of acquisition triggers a stock trading algorithm to buy the stock before the increase in price happens.
Another application of sentiment analysis is monitoring and measurement sentiment for social media posts. During the announcement of Brexit, a social media sentiment tool predicted that “remain” polls were incorrect, as much as six hours before the news broke.
Image source: Techcrunch
Enhancing customer experience
Sentiment analysis can be used to focus on the customer feedback verbatims where the sentiment is strongly negative. Likewise, we can look at positive customer comments to find out why these customers love us. Only after these sentiment analysis have been conducted successfully, we can focus on increasing the number of our promoters.
When used in combination with Thematic analysis, we can further narrow down this information to find precisely which themes are talked about with positive/negative sentiment. This provides actionable insights for your business.
Evidently, sentiment analysis is being used by taking a source of text data that has a narrow scope of context and then gauging the polarity of the text.
Two basic techniques for sentiment analysis
1. Rule-based sentiment analysis
The first technique is rules-based and uses a dictionary of words labelled by sentiment to determine the sentiment of a sentence. Sentiment scores typically need to be combined with additional rules to mitigate sentences containing negations, sarcasm, or dependent clauses.
Sentiment Dictionary Example:
-1 = Negative / +1 = Positive
2. Machine Learning (ML) based sentiment analysis
Here, we train an ML model to recognize the sentiment based on the words and their order using a sentiment-labelled training set. This approach depends largely on the type of algorithm and the quality of the training data used.
Let’s look again at the stock trading example mentioned above. We take news headlines, and narrow them to lines which mention the particular company that we are interested in (often done by another NLP technique, called Named Entity Recognition) and then gauge the polarity of the sentiment in the text.
One way to make this approach fit other types of problems is to measure polarity across other dimensions. You could look at specific emotions. How angry was the person when they were writing the text? How much fear is conveyed in the text?
Advantages of using sentiment analysis
By using sentiment analysis, you gauge how customers feel about different areas of your business without having to read thousands of customer comments at once.
If you have thousands of feedback per month, it is impossible for one person to read all of these responses. By using sentiment analysis and automating this process, you can easily drill down into different customer segments of your business and get a better understanding of sentiment in these segments.
Disadvantages of using sentiment analysis
While sentiment analysis is useful, it is not a complete replacement for reading survey responses. Often, there are useful nuances in the comments themselves. Where sentiment analysis can help you further is by identifying which of these comments you should read.
How sentiment analysis actually works
Traditional sentiment analysis involves using reference dictionaries of how positive certain words are and then calculating the average of these scores as the sentiment of that text.
Next is using a simple ML model to make the classification. This is done by generating “features” from the text then using these features to predict a “label”. An example of generating features is splitting the text up into words and then using these words and their frequencies in text as features.
Use ML to assess how positive or negative words are
The “label” will be a measure of how positive or negative the sentiment is. Once the problem has been set up, mathematical optimization techniques are used to create a model. The key difference is ML to assign how positive or negative the features or words are, rather than looking this information up in dictionaries.
The traditional ML techniques are able to obtain reasonable results, but suffer from a problems such as requiring manual work in creating the features. They also do not have a good solution for considering word order. These problems have been addressed by a family of ML techniques known as “Deep Learning”.
Different types of Deep Learning techniques
Deep Learning techniques are also known as Artificial Neural Networks. These techniques have given great advances in Natural Language Processing in the last few years.
One particular model known as the LSTM (Long Short-Term Memory) has been dominating most NLP tasks in the last few years achieving state of the art results. An LSTM approach reads text sequentially and stores relevant information to the task at hand.
Within the LSTM there are cells which control what information is remembered and what is forgotten. In the case of sentiment analysis negation is very important. For example, the difference between “great” and “not great”. An LSTM trained to predict sentiment will learn that this is important and get good at understanding which words should be negated. By reading large amounts of text an LSTM can be thought of as ‘learning’ grammar rules.
Deep learning architectures continue to advance with innovations such as the Sentiment Neuron which is an unsupervised system (a system that does not need labelled training data) coming from Open.ai. Google has developed the Transformer and recently added pretraining (pre-training is where you train a model on a different task before fine tuning with your specialised dataset) to the transformer with a technique known as BERT , achieving state of the art results across many NLP tasks.
Example of how Open.ai’s sentiment predicts sentiment based on characters rather than words.
How to use sentiment analysis for customer feedback
Once you have your sentiment scores, what do you do with them? The simplest implementation is to measure the sentiment across each of your responses and take the average as a gauge for your overall sentiment. Track your average sentiment over time to get a feel for how your customers feel towards your business. From here, you can look at segmenting the data and comparing different segments. For example, if your business operates in different locations, or whether you have some demographic information you can use for segmenting customers.
Further, you can use a text analytics solution, such as Thematic, to further split up the sentiment across different themes found in your data. (And PS. here’s why Thematic analysis, if accurate, always beats simple sentiment analysis).
Options for applying sentiment analysis to your business
There are many APIs available and it can be a good thing to try different techniques. The downside with this is that you will need to set it up yourself and provide all of the infrastructures around it. In addition, these sentiment tools are generalised across many different types of text and document data and not specific to customer feedback.
Sentiment analysis tools
There are various tools on the market for text analytics and sentiment analysis. At Thematic, we’re focused on staying up to date with the latest NLP research and the most successful models used in academia, where there has been a huge amount of progress in the last 4-5 years. Our team at Thematic implements these models and then trains them on a specific dataset for customer feedback. Thereby, we can create a reliable, and accurate analysis for our clients.