Why Does Sentiment Analysis Disagree With the Star Rating on the Same Review?

It usually surfaces during an evaluation. Someone pulls up a 3-star review, sees the analytics tool tagging parts of it as positive, and asks the obvious question: is this thing broken?

It isn't, and the disagreement is the point. A star rating is one number covering a whole review. Theme-level sentiment is scored per statement inside that review. A 3-star review that says "delivery was fast, but the price is hard to justify" genuinely contains positive sentiment about delivery and negative sentiment about price. Thematic scores those two statements separately and attaches each to its own theme. It traces both back to the exact words that produced them. Collapsing that into one label to match the star rating would destroy the information the review actually carries.

Two things are worth saying up front. Ratings and text sentiment agree most of the time, so disagreement is the exception rather than the norm. And there are cases where the star rating is the better number to trust. Below is what each measurement is, why they come apart, when to believe which one, and what to ask a vendor in a demo.

What the two numbers actually measure

They look comparable because both are numbers attached to the same review. They aren't measuring the same object.

• A star rating is a single overall judgment. The customer compresses everything they thought into one value on a five-point scale. Everything below that compression is lost.
• Document-level sentiment is a single computed judgment. A model reads the whole review and returns one label. In the research literature this approach assumes "each document expresses views on a single entity," which is why it "is not applicable to documents which evaluate or compare multiple objects."
• Theme-level sentiment is many judgments. Also called aspect-based sentiment analysis, it pairs each topic in the text with the opinion attached to it, then scores each pair. One review can produce five sentiment values, and they can point in different directions.

If a tool reports theme-level sentiment and you compare it against the star rating, you're comparing five numbers to one. They were never going to match. The only comparison that means anything is document-level sentiment against the star rating. Even that one comes apart, for reasons worth understanding.

The star rating was already a lossy summary

Before any model touches the text, the rating itself carries a known distortion.

Research by Nan Hu, Paul Pavlou, and Jennifer Zhang drew on Amazon data covering 32,878 book titles, 17,978 DVDs, and 28,983 videos. They limited the analysis to products with at least twenty reviews each. Across those, "almost all products have an asymmetric bimodal (J-shaped) distribution with more positive than negative reviews." Ratings pile up at five stars and again at one star, with a dip in the middle.

The reason is the striking part. When the same researchers ran an experiment in which every respondent had to write a review, the ratings came out "approximately normal" with "roughly equal number of positive and negative reviews." The J shape isn't how people feel. It's who chooses to speak.

They named two causes. Purchasing bias: "only consumers with favorable disposition towards a product purchase the product and have the opportunity to write a product review." And under-reporting bias: "consumers with polarized (either positive or negative) reviews are more likely to report their reviews than consumers with moderate reviews."

That second one matters here directly. Moderate opinions are underrepresented in star ratings. So when a 3-star review does appear, it's often written by someone with a genuinely mixed view who bothered to explain it. Those reviews carry more usable detail than the rating scale can represent, which is exactly when a single label fails hardest.

What a disagreement is actually telling you

Researchers at the University of Maryland, Baltimore County, and Wilfrid Laurier University analyzed 24,539 Yelp reviews and concluded that "the inconsistency between review ratings and review content really exists." Their other finding is the one to hold onto: "the ratings and sentiments are highly correlated."

Both are true. The numbers usually agree, and the exceptions are informative. The paper doesn't categorize the exceptions, but in practice four cases account for most of what teams run into.

Only rating inertia is a data-quality problem. The other three are signal the rating can't hold.

When the star rating is the better number

This part matters, because a vendor who claims their sentiment score always wins is overselling.

Use the star rating when you need a single, stable, comparable metric over time or across products. It's the customer's own summary judgment. It needs no model, and nobody argues about its methodology in a board meeting. For tracking a headline number quarter over quarter, it's simpler and sufficient.

Use theme-level sentiment when you need to know why the number moved, which specific issues are driving it, and what to fix first. The rating tells you the score changed. Only the text tells you what changed.

The failure mode is picking one and discarding the other. The rating is the outcome measure. The themes are the explanation. A program that reports only ratings can't act, and a program that reports only sentiment can't prove impact.

How Thematic scores sentiment on the same review

Sentiment is attached to phrases, not to reviews. Thematic identifies meaningful phrases in the text and groups them into themes. It assigns each theme a sentiment of positive, negative, or neutral. A single comment can carry several themes with different sentiment on each.

Neutral is a real category, not a rounding error. Statements that don't lean either way are scored neutral rather than being pushed toward positive or negative. That's what keeps a mixed 3-star review from being flattened into one misleading label.

Every score traces back to the words that produced it. When a reviewer asks why a passage in a 3-star review was scored positive, Thematic shows the phrase, its theme, and its sentiment. The answer to the "is this broken" question is a drill-down, not an argument.

Themes can be tracked against the rating rather than compared to it. Thematic tracks the rating over time as the outcome measure, then identifies which themes moved alongside it. That treats the two numbers as complements, which is what they are.

A worked example: half a star, one cause

Thematic ran this on public data and published the result. Analyzing over 10,000 reviews of the Apple Music app on the Google Play store, the analysis found that the Android app's star rating dropped by roughly half a point over a month in August 2021.

The rating showed the drop. It said nothing about the cause. The theme driving it was "downloading," with more than 200 individual reviews mentioning it. Inside that theme, customers said new songs wouldn't download and that individual songs cut out. Albums and playlists were unaffected. The theme's volume fell after the underlying bug was fixed.

That's the relationship in one example. The rating was the alarm. The themes were the diagnosis. Neither would have been enough alone. (The analysis dates from 2021; the mechanism it illustrates has not changed.)

What to ask in a vendor demo

Run these against your own data, ideally on a set of reviews you already know well:

1. Show me a 3-star review and every theme and sentiment value the tool assigned inside it.
2. Can you trace each sentiment value back to the specific phrase that produced it?
3. Is neutral a distinct category, or does the model push everything toward positive or negative?
4. Can the tool report theme sentiment and the star rating side by side over the same period?
5. When a rating moves, can the tool tell me which themes moved with it?
6. How does the tool handle a review that praises one aspect and criticizes another?
7. What happens to reviews where the rating is stale but the text is recent?

If a vendor answers question one by showing a single sentiment score for the whole review, that tool is doing document-level analysis, and it will keep disagreeing with your ratings in ways nobody can explain.

The short answer

Sentiment analysis disagrees with the star rating because they measure different things at different resolutions. The rating is one overall judgment for a whole review. Theme-level sentiment is a separate judgment for each thing the review talks about. A mixed review is supposed to produce mixed sentiment, so a 3-star review with a positive comment about delivery is being read correctly, not incorrectly.

Ratings and sentiment agree most of the time. Treat the disagreements as the reviews worth reading, because they are the ones where the customer said something the scale could not capture.