You've successfully subscribed to Thematic
Great! Next, complete checkout for full access to Thematic
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

What is Deep Learning? A brief history, and what it means for the future

At Thematic, we have a dedicated research team that is always innovating to bring you the most accurate sentiment and thematic analysis. Solving this difficult task requires in-depth understanding of both the problem of customer feedback analysis and the relevant advances in technology.

This article is a glimpse into one of the technologies we use here, called Deep Learning. Knowing the technical details of Deep Learning is not required to be able to see the power of its capabilities and what it will bring in the future.

Deep Learning: What is it?

Let’s start by getting the technical details out of the way...

Machine learning diagram
Image credit

Deep Learning is a form of Machine Learning. It is known as 'Deep' Learning because it contains many layers of neurons. A neuron within a Deep Learning network is similar to a neuron of the human brain - another name for Deep Learning is 'Artificial Neural Networks'.

Diagram of a biological neuron
Diagram of a biological neuron Image credit

A Deep Learning model is trained by setting an objective for what the model wants to learn, then optimizing the model’s parameters to improve performance. This is different to traditional Machine Learning, which requires manual crafting and selecting of features (properties of data that the algorithm should examine) which takes time and is not always effective.

Understanding how Deep Learning works is not important for this article, but if you would like a crash course on the details, have a look at this video: But what is a Neural Network?

The History of Deep Learning

Below is a timeline of some of the important events in the history of Deep Learning. They are by no means exhaustive, but they are chosen as they help illustrate how Deep Learning arrived to where it is today.

Deep Learning Timeline:

  • 1958: Perceptron (Simple version of deep learning) Discovered
  • 1979: Convolutional Neural Network Invented (Today dominates image recognition)
  • 1982: Recurrent Neural Network Invented (Sequence processing used for NLP)
  • ❄️ AI WINTER ❄️
  • 2009: ImageNet Introduced
  • 2010: Kaggle Launched

The first three events show when some of the techniques that are used today were invented. At the time, these techniques were unable to be used for real world applications. This was largely because computing power was hard to come by and some of the details that make these techniques effective were yet to be discovered. An interesting point to keep in mind is that these technologies are not new ideas, but applying them effectively to real world problems is relatively new.

While the exact date of the beginning and end of the AI winter is debatable, there is no question that there was an AI winter, and it was icy. Then, sometime around 2010, the frost began to melt. The reason I have included ImageNet and Kaggle on this timeline is that they were instrumental in the recent progress of Deep Learning, and the defrosting of the AI winter.

What is ImageNet and how did it help to advance Deep Learning?

ImageNet is a large dataset which is used for image classification. Image classification is the problem of being given an image, then having to label it with which predefined category it belongs to. An example of this is a classifier that must determine if each image is a cat or if it is a dog.

Screenshot from the dogcat image classification model
Image credit

Another example of this, (especially relevant to those who have seen the TV show Silicon Valley) is the “hot dog / not a hotdog” classifier:

Image from the hot dog / not a hot dog classifier, comparing a hot dog to a shoe
Image source

ImageNet takes this task of image classification but extends it to over 20,000 different classes, with an average of more than 500 examples of images from each class. Below are some real examples from the dataset.

Dataset examples from ImageNet
Image source

What effect has ImageNet had on Deep Learning and AI?

  1. Provided a measure to benchmark how good a technique is
  2. Provided an arena for competition between different approaches

Much like how the UFC (Ultimate Fighting Championship) enabled different styles of fighting and martial arts to compete against each other, ImageNet has meant that different techniques of image recognition can be compared and the ones that work the best can rise to the top.

Here is a chart of the state of the art accuracy scores over the last ten years:

Graph of accuracy of image classification on ImageNet

Over the last decade, accuracy on this task has improved from getting around half of the predictions correct, to getting 9 out of 10 correct, which is widely accepted as beyond human accuracy.

The advancements along the way represent discoveries of small tweaks to Neural Networks which allow them to work much better. It is important to note that the performance on ImageNet is still increasing at a rapid pace.

What is Kaggle?

In the timeline, we also have the founding of Kaggle in 2010. Kaggle is a platform which hosts competitions which are similar to ImageNet, but over a wide variety of applications, from images, to tabular data, language and many areas. Kaggle has had a similar driving force effect on the progress of Deep Learning.

What Natural Language Processing (NLP) shares with Image Processing

What is Natural Language Processing?

Natural Language Processing (NLP) is the intersection of Computer Science and Linguistics. For example, could an algorithm read this sentence and understand what it means? Well, current NLP technology is able to extract themes from sentences and deliver humans insight into what is being said. Perfect for analyzing large amounts of text!

At Thematic, we use NLP tech to analyze and find insights in large amounts of customer feedback for our clients.

Why is all of this information about image recognition and ImageNet relevant when at Thematic we live and breathe NLP?

Firstly, ImageNet has been the pivotal dataset when it comes to the progress of Deep Learning so it is a good one to look at. Secondly, progress with NLP tasks has been following a similar path to progress with images although it is around 2 or 3 years behind. This is because most of the techniques discovered for images are transferable to be used with language (and vice versa). NLP is lagging by a few years because in some ways it is more difficult to work with language.

You can think about both images and language in terms of the compression of information. For example, you can have a high resolution image and a longer sentence representing the same thing as in a low resolution image or a short sentence. Because of all of the shared context we have around words and meaning, we are able to compress a lot of language information into a smaller space (a sentence).

This contrasts with an image, if we look at a very small image like an icon on the iPhone, which is 120x120 pixels. We require 14,400 pixels to represent something which we could describe in one, or very few words.

iPhone screen showing several app icons
Image source

Narrow vs General Intelligence: Will AI take over the world?

Many people, including smart people and celebrities, are concerned with the dangers that AI brings. People who actually work in AI, don’t tend to share these fears, and here's why:

There is a difference between General Intelligence (the ability to be successful across a wide variety of tasks) and the ability to automate the processing of a very narrow task (such as identify if the image contains a cat or a dog). The majority of Deep Learning research is focused on the latter.

The first part of the Deep Learning revolution has involved getting near human or above human performance on narrow tasks. For example “Is there a stop sign in this image?”

Part of the criticism of recent advances is that if the task is modified slightly, the model won’t work. Model abilities are now getting more general, although still far from General Intelligence as we know it. Extending the stop sign example, the models now can answer questions  like: “How many stop signs are in the image?”, “Draw a box around the stop sign(s) in the image”. And to extend from here, the stop sign detector is now a small part of a wider AI system for autonomously driving a vehicle. If you’d like to learn more about Deep Learning for stop signs and autonomous vehicles I recommend watching this video AI for full self driving.

Conclusion

Deep Learning continues to advance at a rapid pace. This makes it an interesting (and at times difficult to keep up with) field to follow. While one day, these techniques are likely to be surpassed by something new, they are currently the best performing methods for a lot of tasks and are showing no signs of reaching their limits in performance. And as performance increases, we are going to see Deep Learning be involved in more parts of the world we live in.

For Thematic, this means that our ability to determine the sentiment surrounding a particular sentence, part of a sentence or a theme, is continuously improving. By investing in R&D in this field, we are helping more and more companies to discover 'unknown unknowns' in their data and provide an accurate view of what their customers want.

Ready to scale customer insights from feedback?

Our experts will show you how Thematic works, how to discover pain points and track the ROI of decisions. To access your free trial, book a personal demo today.

Recent posts

How to theme qualitative data using thematic analysis software
How to theme qualitative data using thematic analysis software
Members Public

Become a qualitative theming pro! Creating a perfect code frame is hard, but thematic analysis software makes the process much easier.

AI & NLP
Text Analytics Software – How to unlock the drivers behind your performance
Text Analytics Software – How to unlock the drivers behind your performance
Members Public

Many organisations, large or small, gather customer feedback to improve their CX efforts and ultimately their bottom line. But gathering feedback alone can’t make much of a difference. We need to analyze our feedback to discover insights that inspire us to drive action at our organisations. Enter; Text Analytics

AI & NLP
Thematic Analysis Software: how it works and why you need It
Thematic Analysis Software: how it works and why you need It
Members Public

With AI-driven thematic analysis software, you can generate actionable insights effortlessly. Here’s what you need to know about this method of data analysis tool.

AI & NLP