Sarcasm in customer feedback – how common is it?

Most people believe that text analytics solutions fail because sarcasm in customer feedback is very common. Somebody writes “Great service, yeah right!” and the dumb algorithm tags it as positive. So, whenever I speak on text analytics, someone in the audience will always ask:

But how do you deal with sarcasm?

People tried writing complex algorithms to detect sarcasm and irony. But are there practical solutions that work? And, more importantly, do we actually need them? How common is sarcasm in customer feedback?

What is sarcasm?

First of all, what is sarcasm? Sarcasm, in simple terms, is when someone speaks bitterly or contradictory to the true intention. Turns out, there are seven types of sarcasm. Sarcasm is also an example of a verbal irony, something that occurs when the underlying meaning is contradictory to what is being said. There are other examples of verbal irony such as overstatement and understatement.

A classic example, at least here in New Zealand, take the Tui Brewery’s Yeah Right billboards (see the image above). Sarcasm can be challenging, misleading and annoying. I am an immigrant myself, submerged in the British culture. So, I can attest that it will take me many more years before “I get it”.

So, my British business partner sent me this…

Sarcasm, Algorithms, Customer Feedback, and Your Brand

Can algorithms detect sarcasm?

Understanding sarcasm and irony is hard. Just like understanding humor, it requires not just language proficiency, but also knowledge of common sense, cultural knowledge, and context. These can be quite intricate.

Unfortunately, we have not yet succeeded at teaching computers common sense, nor cultural knowledge. Capturing context is another hard nut to crack. This means that unless there are clear clues of irony such as emoji or heavy punctuation, today’s text analytics will struggle with sarcasm.

But how common are sarcasm and irony in customer feedback? And how useful is it for brands being able to deal with them correctly?

How common is sarcasm in customer feedback?

I recently spoke to people at two large Australasian retailers. Both stated that Kiwis and Australians love sarcasm. They were convinced that their customers leave plenty of sarcastic comments, both in surveys and on social media. I was curious if that’s true.

I reviewed two datasets with customer feedback by hand to figure out how common sarcastic sentences were:

• 500 survey responses
• 500 posts on social media

Turns out, even in retail, less than 1% of survey comments and less than 5% of social media comments contained sarcasm. This corresponds to findings in other datasets. The training dataset for sarcasm detection competition at PAKDD 2016 also contained only 3% of positive examples.

But the fact that sarcasm is rare is just one part of the story. Let’s look at the examples of sarcasm I found.

When is sarcasm useful to brands?

Somebody wrote on Facebook: “Very helpful answer. Troll.” Troll, in the context of social media, is a derogatory term, so the comment is sarcastic. The answer wasn’t helpful at all. Most algorithms would fail to recognize this.

But so what? It’s just some banter on social media. Half of the sarcasm examples I found were of this type: meaningless back and forth in comments to posts unrelated to the actual topic of the conversation.

What brands want to know is what people are specifically saying about the product, the brand, competition, or people in customer-facing roles. Detecting sarcasm in people’s reaction to each other’s opinion is not of much value. This brings down the percentage of meaningful sarcasm even lower.

Most other examples of sarcasm were part of a longer comment. For example, “I’m so disappointed! What a great customer service you have!” It’s impossible for a person to be disappointed at and praise something at the same time. Hence, we can conclude that it’s a sarcastic comment. The service wasn’t great at all. This is an example of where the brand will be interested in this information.

The good news is that most algorithms looking at the whole comment will mark the general sentiment as either negative or neutral. A useful algorithm will also detect “customer service” as the topic of this comment. Following that, people’s sentiment will be aggregated across all comments correctly.

Conclusion: Remember this about sarcasm in customer comments

• Sarcasm in customer feedback is difficult to understand for an algorithm. But it is incredibly rare! Rarer than you think!
• Most examples of sarcasm on social media aren’t tied to any specific topic. Therefore, they are meaningless to brands.
• Sarcasm in relation to a specific theme in customer comment is likely to appear alongside other indicators of the author’s true sentiment. Algorithms are likely to identify correctly those.