TL;DR: Unsupervised anomaly detection provides a way to detect product defects without the need for excessive data collection and data labeling of all possible defect types. Unsupervised anomaly detection methods are generally divided into feature embedding-based methods and reconstruction-based methods, each having their pros and cons. Several commercial and open-source solutions, as well as some benchmark datasets, exist to get started.

Despite the effectiveness of unsupervised anomaly detection methods, one should be thoughtful about some common pitfalls like the prediction of many false positives and not having the identification of the detected defects. Some good practices to improve performance and robustness are eliminating product variation and background clutter, having tailored hardware that accentuates the defects, or training a supervised anomaly detection system based on the output of the unsupervised system. 

A defect in your production process?

In the manufacturing and food industries, ensuring the quality of products is of paramount importance. Defective products should be identified as fast as possible in the production process to prevent scrap, rework, or to prevent them from reaching customers, as this can lead to costly claims and brand damage. Product defects of all kinds can occur in the manufacturing or food industry, such as color deviations or holes in textile products, scratches or dents in metal or wooden products, foreign objects in food products, burned fibers or sticky patterns in polymer products, etc. 

In today’s production plants, operators may still perform manual visual quality inspections of product samples on the production line. This in order to remove a product from the production line or even to shut down the line if a defect occurs. This manual intervention is highly impractical because products can pass at high rates, there may be too many production lines to be able to be covered completely, or product defects can be subtle. Additionally, manual visual quality inspections by operators are inherently subjective and not operative 24/7 at optimal performance. Estimates vary, but in some applications, there can be seen that only 70% of product defects are identified by manual inspection, leaving a significant 30% undetected when reaching the customer.

Figure 1: Sample from the MVTech dataset that shows some of the product categories and product defects that are within this dataset.

What about automated visual inspection?

To overcome the limitations of manual visual inspection, a more robust, objective, and reliable inspection performance can be obtained via automated visual inspection. Quality inspections can be executed inline using cameras and proper software, and every single product can be inspected consistently and continuously. However, visual defect detection using computer vision is a challenging and often application-specific task. Sometimes, tailored hardware is required to reveal the defect in high contrast in an image, and the software needs to be highly performant to detect possible subtle defects at high detection rates without many false positives. The research towards visual defect detection is vast and many methods exist that each have their own properties and that can offer an added value in plenty of specific applications. 

Figure 2: Image of an automated visual inspection system

So now collect as much data as possible and train a model on it? 

In some cases, using specific hardware and under constant environmental circumstances, product defects could be detected robustly using traditional image processing techniques. If possible, it should always be considered before diving into AI solutions. However, classical non-data-driven techniques may not be adequate when products are more complex (inherently varied like food or natural products), or defects are diverse and subtle. For more complex situations, the current state of practice is highly focused on the supervised learning of neural networks, which relies on labeled data to learn the model to predict whether a product is defective. 

Supervised anomaly detection requires a collected dataset to train a model until it is sufficient to work in production. The dataset should contain images of all possible cases that may occur, meaning images of all types of defects and also images of non-defective products. This may be challenging in practice as it may take a lot of time and effort to label. If data were collected continuously for a long time, one would end up with a highly imbalanced dataset that mainly includes images of non-defective products and only a small portion of defective data, where some defects may be underrepresented or absent. In some cases, even only one defect out of 100.000 products needs to be detected. Labeling so much data with a majority of irrelevant, non-defective images is tedious, and training a supervised model on this unbalanced labeled data would result in poor system performance. The model would also not be able to robustly detect defects that were not seen before. Artificial defects may be used, but this may distract the model if the artificial defect does not align with reality.

The unsupervised way

Unsupervised anomaly detection is gaining popularity due to the assumption that the collection of defective samples is costly. Unsupervised anomaly detection removes the need for having a complete labeled dataset of all defect types and can already be trained on only a small number (10-100) of non-defective samples. But it should also be mentioned that it often comes with challenges in situations where there is a large product variety (different colors, patterns, illumination, etc.) or when defects are subtle. In general, unsupervised anomaly detection models can be subdivided into feature embedding-based methods and reconstruction-based methods.

Feature embedding-based methods 

Explanation: Feature embedding-based methods use pre-trained convolutional neural networks (CNN) to create embeddings (feature vectors) of images or image patches. During training, embeddings of only non-defective samples are created to form a ‘normal’ reference in the form of a Multivariate Gaussian Distribution or a simple memory bank of all retrieved normal patch embeddings.

During inference, image embeddings are created for each new image and the system calculates how ‘far’ the embeddings of the new image are from the normal distribution. The calculated distance is referred to as the anomaly score and can be thresholded to determine if the new image is anomalous or not, i.e., if it contains a defect. Other feature embedding-based methods exist that use a teacher-student architecture where the knowledge of a pre-trained teacher network (e.g. pre-trained on ImageNet) is distilled by a smaller student network due to the student network learning to predict the same image features for a non-defective image as the teacher network predicts. The detection of anomalies during inference is enabled by the student failing to predict the anomalous features as it is trained to only predict features of non-defective images.

Pros

Feature embedding-based methods:

• (i) are able to detect subtle and complex anomalies as the used pre-trained models can usually extract highly informative features;

• (ii) can be constructed quickly as constructing a Multivariate Gaussian distribution or a memory bank is a lightweight process. This makes it suitable for quick retraining on a new dataset for e.g. other products;

• (iii) are typically light weight for inference as there is only a need for checking if the feature vector of a new image lies in the trained ‘normal’ distribution or not;

• (iv) are most suited for image-level anomaly detection as they can extract high-level features from the images.

Cons

Feature embedding-based methods:

• (i) have a performance that is highly dependent on the underlying pre-trained network and that can be low if they were pre-trained on data (often ImageNet) that significantly differs from the product data. Fine-tuning the pre-trained network on the custom product data may be needed;

• (ii) are not very suited if the underlying product shows a large variation as they can only train one distribution. The imaged product should therefore be as consistent as possible; 

• (iii) lack pixel-level anomaly detection as they are based on retrieving feature embeddings calculated on image patches or even on the full image, but have no pixel-level information. 

Popular models

Some popular state-of-the-art models using feature-based methods are PaDiM, which uses a Multivariate Gaussian Distribution to get a probabilistic representation of the normal class; PatchCore, which uses a simple memory bank as a normal image distribution; and EfficientAD, which is based on a teacher-student architecture and is therefore very lightweight, computationally efficient, and suitable for real-time applications. 

Figure 3: Illustration of feature embedding-based methods where image features are extracted using a pre-trained backbone and mapped onto a Gaussian distribution.

Reconstruction based methods 

Explanation: Reconstruction-based methods use encoders and decoders to reconstruct images for anomaly detection. During the training process, normal images are sent to the reconstruction network which learns how to encode the images into a latent space (space that contains the core information of the image in a much lower dimension) and decode it again to reconstruct the original images.

During inference, when given a defective image, reconstruction networks tend to result in an image with no defect as they are only trained to reconstruct normal images. A comparison model compares the original image to the reconstructed image to calculate the reconstruction error which is further referred to as the anomaly score which can be thresholded to determine if the new image is anomalous or not, i.e. contains a defect or not.

These methods can also provide pixel-level anomaly detection, which outputs a segmentation map highlighting the exact location of the defect. Several methods can be used as a reconstruction backbone, such as autoencoders, GANs, Transformers, and diffusion networks. 

Pros

Reconstruction based methods:

• (i) are not reliant on any pre-trained model which increases their ability to detect anomalies. Instead, their performance varies depending on the reconstruction model and abnormal sample construction methods;

• (ii) are performing well for pixel-level anomaly detection as they enable pixel-level comparison.

Cons

Reconstruction based methods:

• (i) can be computationally expensive, especially for high-resolution images, due to the reconstruction network;

• (ii) might overfit the training data and learn to reconstruct also the anomalies. This prevents it from detecting the anomalies as the reconstructed network then shows no difference (low anomaly score) with the original defective image;

• (iii) have poor image-level anomaly detection capabilities due to their inability to extract high-level semantic features. 

Popular models

Some popular state-of-the-art models using reconstruction-based methods are RealNet, DiffusionAD, and Denoising Diffusion Anomaly Detection (DDAD).

Figure 4: Illustration of reconstruction-based methods where an encoder-decoder architecture aims to reconstruct the original image. 

I’m convinced! Now how can I put this into practice?

If you want to play around with unsupervised anomaly detection in a no-code and user-friendly way, you can consider commercial solutions such as the Red Analyze tool of the Cognex VisionPro software suite, the Halcon Anomaly Detection tool from MVTech, and the Keyence VS Series with anomaly detection functionalities. These systems get you started pretty easily without required coding experience, are very user-friendly, and can provide sufficient results for your application.

However, these come at relatively high costs compared to open-source solutions and are often unsuitable for more complex and tailored cases requiring high accuracy and low false positives. If you have programming experience and a use case that requires more customized data modeling, you can also consider open-source solutions. Most of the existing models have a separate code repository which can be used to implement the model in your own code. But when you want to explore several models, implementing each of them separately can be tedious as they are all built differently.

A useful and popular framework for unsupervised anomaly detection exists and is called Anomalib. This framework offers an easy-to-use implementation of several state-of-the-art anomaly detection algorithms and also contains some frequently used datasets that can be used to benchmark existing or custom models.

Some public datasets for anomaly detection that can get you started if you have no data yet are MVTecAD, One-Class CIFAR-10, NEU Surface Defect Database, and MVTecLocoAD. The MVTecAD dataset is used a lot in literature for benchmarking anomaly detection methods with a focus on industrial inspection. It contains over 5000 high-resolution images divided into 15 object and texture categories. Each category comprises a set of defect-free training images and a test set of images with various kinds of defects as well as images without defects.

Common pitfalls and good practices to overcome them

Beside its efficiencies and the ability to train a model without labeled data, unsupervised anomaly detection has some limitations that complicate its implementation. One of the main pitfalls is that unsupervised anomaly detection models typically tend to result in many false positives which may not be interesting for a company if detecting an anomaly would mean that a machine should be shut down. False positives can occur when the underlying product shows some variation that can be seen as being anomalous with respect to the normal data. Also background clutter or subtle variations in the background can have this effect. Some good practices can be taken to overcome this:

• First, to remove product variation, one can make sure that the product to inspect is always in the same position within the image. To do so, it can be useful to have an object detector before the anomaly detector that detects the object and that can send only the object ROI to the anomaly detector. The object detector does not have to be a neural network but can also simply be some image processing techniques to remove backgrounds if possible. To further reduce the product variation in the image, also the environmental circumstances could be kept as constant as possible by using constant illumination within a controlled environment.
  
  

• Second, one can also take into account the importance of having tailored hardware for the defect detection system. People often tend to collect visual data using simple and cheap hardware while relying on AI to solve the problem disregarding the image quality. Using the correct illumination/lens/sensor configuration, some defects may be visible much more clearly in the image making them more easy to detect for successive software systems. Using high-resolution cameras and lenses; dark field illumination to detect scratches or dents; or photogrammetry using the combination of images under illumination from different angles could provide more enhanced images of defects, which will result in more robust detections.
  
  

• Finally, an approach to tackle a potential lower performance of unsupervised anomaly detection systems is to train a supervised anomaly detection system using the output from the unsupervised system. We initially would use an unsupervised system because we do not have defective data yet, but after we have due to the unsupervised system, we could use this data, together with some non-defective samples (which are abundant) to train a supervised defect detector with increased performance. This method has the benefits of having a high-performant supervised detection system but without the lack of being trained on an unbalanced dataset because of the pre-filtering of data by the unsupervised anomaly detector. 

Another drawback of supervised or unsupervised anomaly detection systems is that they only provide a binary ‘good/bad’ decision. This means that they are not informative on the type of defect detected. However, this may be important in practice as a company may be required to act directly upon some defects occurring in production. Some defects may require a shutdown of the machine to avoid excessive amounts of waste. Other defects may only require removing a specific product from the production line.

To mitigate this, a successive stage could be added to the anomaly detector that gets trained on the anomalous samples in a supervised way with defect type labels annotated by an expert. This two-stage approach is much more efficient than a fully supervised approach as all the irrelevant non-defective data is filtered out for labeling by the unsupervised anomaly detector. The latter also prevents ending up with an unbalanced dataset. 

Conclusion 

Unsupervised anomaly detection can provide an efficient way to detect product defects in manufacturing or food industries without the need for thorough data collection and labeling. Two main methods exist that either use feature embeddings to learn a ‘normal’ distribution or use a reconstruction network to reconstruct the ‘normal’ images and find differences with the original defective images. Despite unsupervised methods being convenient in several cases, one should be thoughtful about the common pitfalls and carefully design both the hardware and the software setup to obtain proper results. 

At Superlinear, we are aware of the most recent advances in this field and about the proper actions that can be taken to overcome the common pitfalls. Our team can help design both hardware and software setups to optimize your anomaly detection system for any production environment. Do you want to discuss this topic further, or do you have a case for us? Feel free to reach out!