Scale or fail: Building an infinite customer feedback loop with AI

Atlassian is a leading software company creating tools that enhance collaboration, productivity, and ultimately customer experience. You may know Atlassian best through our flagship Jira brand, Confluence, and Trello.

We serve professionals in software development, IT management (think raising a ticket with IT) and other project management needs.

We have over 250,000 customers who use our products. And the amount of feedback we get is phenomenal. In this post, I'll share our story about the enrichment and automation of Atlassian customer experience.

I will lay out the issues that Atlassian was facing, and explain why and how Thematic has been the right partner to solve these problems.

An overwhelming amount of user feedback

To contextualize Atlassian's experience, think of recurring qualitative feedback your company might receive. This could be recurring surveys you send, in-product feedback or support channels.

The Atlassian example might be an extreme case of feedback scale, but I think the underlying issue applies to many companies.

Traditionally, Atlassian relies on word of mouth and viral products, instead of traditional enterprise sales.

Because of that, we have built a close relationship with our community of users. Our users are seriously passionate and vocal when it comes to our products. There’s no shortage of feedback or suggestions about how to improve experience to support our products.

• Atlassian customers are super engaged, and we’ve fostered a culture of conversation.
• We launched our community to provide a platform where our strongest customer advocates could work with us to support others at scale.
• We encourage our product teams to reach out for customer input and feedback.
• And we established a customer listening program that allowed us to keep our finger on the pulse of what customers value.

Our research and product teams were often overwhelmed by feedback

So while all this was great early in our product maturity, and it is still great, Atlassian’s research and product teams often became overwhelmed by the volume and complexity of the feedback coming in.

They wanted to do the right thing, but didn't know how to engage with, analyze, and fully comprehend feedback effectively at scale.

This leaves users frustrated, not just by the lack of response, but by the lack of change in our products, which is ultimately what they want. This impacts our user experience and overall product satisfaction scores.

Our teams were desperate for the insights to effectively improve customer experiences.

There’s just too much data coming in, and no way to fully understand it.

Our manual feedback solution couldn't scale, and wasn't effective

Our research team would spend 6 weeks in a Miro whiteboard bucketing and categorizing feedback.

This is just time spent on pulse survey data alone (representative sample of our user base), which is less than 20% of the qualitative data we receive.

This would produce a report, on a quarterly basis, which is just too slow in our competitive world. This solution for analyzing feedback isn’t scaling, and it’s also biased to survey responses. Customer experience suffers, and that shows in our CSAT scores.

What Atlassian needed to solve the feedback challenge

Now do we make or buy a solution? It’s not our expertise, we build collaboration and productivity tools.

We needed:

• Quality theming and sentiment analysis
• Someone who could handle our scale
• A solution that could meet our need for real-time theming
• An intuitive way to build and manage the themes
• A way to keep it lightweight and flexible

So, we needed someone who could do a few things really really well, and would be easy to integrate with.

We didn’t need or want an all-in-one solution.

Ultimately, Thematic became our solution and I’ll dive into the reasons why.

We have feedback coming from a lot of different places. We needed to build a real-time pipeline so we could close loops where needed.

We didn't want to send all of our data into an all-in-one solution.

Thematic’s API enabled us to manage complexity across channels quickly and iteratively.

A huge differentiator for Thematic is seamlessly training the model under the hood (if you need to).

Since our use case involved responding to customers based on what Thematic interpreted them saying, we wanted err on the side of accuracy over coverage.

We’d rather hit one target and respond to that, instead of over tagging and responding on something unrelated.

In other words, our risk tolerance was low and we needed control of training this model from a low-tech perspective.

The amazing thing about teaching the model once is that it learns to think like you.

So what might seem like a lot of effort up front, then results in low maintenance costs.

When feedback doesn't fit,  this becomes a rich source for new theme discovery. The AI performs consistently and there's no bias creeping in from different people doing the analysis.

Thematic ticked all the boxes, so we could deal with feedback faster, and more effectively than ever before

Thematic promised this, and I wouldn’t still be working with them 18 months later if those promised weren’t upheld.

Thematic was already doing AI, and they’ve moved quickly to Generative AI solutions that we’re excited to leverage.

In other words, Thematic ticked the boxes. Now, I’ll summarize visually what the problem and solution looked like.

Massive amounts of feedback coming in per month, for a flagship brand like Jira.

Imagine sitting at your digital whiteboard, let alone a physical one, with just 1,000 pieces of feedback…

CSAT survey is our pulse on customer satisfaction, and one of the channel we were aiming to respond to. This is where our research teams were spending up to 6 weeks just bucketing feedback.

Given the maturity of Jira Software, the core customer complaint themes aren’t changing often. So doing this quarter over quarter becomes repetitive and costly, really just to keep quantifying issues. Important stuff, but perfect for a machine to learn.

Imagine spending just a few of those weeks analyzing feedback the machine didn't catch, as a potential for new and emerging themes.

Thematic is your own personal analyst

Well, now we can. Thematic is your own personal analyst, only your analyst works 1000 times faster than you and doesn’t slip out early on Fridays

Now we have cross-channel insights feeding a report, and we can leverage this new segmentation to send more personalized responses to users across the company.

This segmentation is being leveraged by our product research teams when they look for candidates to interview.

An example is listening to how our users are explaining what they expect from AI inside of our products. Customers explain how they wish they could interact with our products.  Using Thematic, we uncovered how users expected to be able to use AI for adoption, bringing new users up to speed faster. And JQL - querying language for finding issues was something users had to learn, and now they expect to find issues using natural language.

In a fast-paced tech landscape, I have high expectations for the customer experience.

The thing I love most about working with Thematic is that they go the extra mile to be supportive and responsive.

Every request and suggestion I have is considered and Atlassian feedback is clearly valued. This enables us to deliver insights and a scalable feedback loop with OUR customers.