Executive summary

Context

Statistics Flanders is the network of Flemish government agencies that develop, produce and publish official statistics. Their goal is to make reliable statistics while utilizing various data sources. They decided to experiment with publicly available data (such as social media data) to see how Artificial Intelligence can contribute to producing reliable statistics.

Goal

To find a more affordable and efficient way of acquiring insights into public sentiment, comparing the use of publicly available data vs. traditional surveys; To accurately assess general sentiment in Flanders to understand citizens' concerns and thoughts. For example, they wanted to understand how much discomfort was sparked by COVID-19 and how it changed over time and how the Ukrainian war made Flemish citizens feel. 

Solution

An AI-powered dashboard displaying public sentiment on various relevant topics (e.g., Ukrainian War, COVID-19), based on analyzing up to 50k Flemish tweets per month.

Case study

The challenge

Statistics Flanders is constantly exploring various data sources to ensure the quality of their work. Usually, they use surveys and data from third parties to produce official statistics, but surveys can only be performed infrequently and are relatively expensive. Statistics Flanders is looking for complementary data and alternatives to surveys in order to increase the frequency and decrease the costs.

Stemming from this, Statistics Flanders wanted to employ new data sources that were more accessible, affordable, and would help the agencies consult citizens directly and fast. Artificial Intelligence can provide the level of insight that data science and statistics need for accurate and complete results.

This work is in line with projects in a number of official statistical agencies worldwide who are already looking into these types of data and methods to evaluate, for example, public sentiment to inform public policy.

For this project, the Statistics Flanders team chose to experiment with Twitter data because: 

• The data is publicly available.

• The Twitter API is easily accessible.

• Twitter has many active users in Flanders.

There were many considerations before starting the work, however, such as: 

• How do we make sure vocal minorities don’t nudge statistics the wrong way? 

• How do we deal with the fact that not everyone uses Twitter and people use it to express more complaints than positive, supportive comments? 

Taking these considerations into account, we got started. 

The work

Here’s what the process looked like:

Using Artificial Intelligence was new for Statistics Flanders, and we needed to make sure their end-users felt at ease using it to analyze what citizens say. The Statistics Flanders team had concerns about how reliable the results would be and how the solution would work. Their team asked Superlinear many questions and they now understand and trust the model.

‘The Superlinear team was very skillful. They used both data engineering and natural language processing. I was impressed by the level of competence, and it was also pleasant to work together. All team members complemented each other nicely - we had a project lead, a solution lead, and the engineering team.’

Michael Reusens - Data Science Coordinator, Statistics Flanders 

The project took two months to complete, with an additional month for integration. We started working on it during COVID, so the project had to be executed entirely digitally. 

‘No hurdles. Superlinear did a great job taking control of the process, proposing a timeline, methodology, and executing it.’

Michael Reusens, Data Science Coordinator, Statistics Flanders

The result

We delivered a complete AI-driven solution that extracts tweets to a dashboard, where users can monitor sentiment day by day or within a specific timeframe. Statistics Flanders is planning to use this data to supplement official statistics and directly listen to citizens' voices.

For example, take a look at the Flemish public sentiment during February 2022. You can see a clear spike in negative sentiment on the 24th of February when the Ukrainian war started. You can also observe a slight spike in positive sentiment at the same time. Could it be connected to supporting Ukrainians? Perhaps.  Our dashboards accurately show Flemish citizens’ sentiment on February 24th. Now, the team of Statistics Flanders can focus on using their knowledge and skills to interpret these sentiments.

Currently, Statistics Flanders is experimenting with this dashboard to see how to provide maximal value and accuracy to the public. They are running several research tracks, such as bias in population, bias in annotation, and optimal machine learning models to ensure they release statistics that are as accurate and unbiased as possible. 

Application to other industries

You can use sentiment analysis to assess public opinion about virtually anything. Here’s what you can do with it:

1. Keep track of how customers feel about your products or services.

2. Identify the most problematic areas of your business by analyzing customer feedback and reviews.

3. Analyze the distribution of positive and negative feelings about your brand.

Sentiment analysis is not limited to only text input (social media posts, email, written reviews etc.). You can also make use of video and audio, depending on which source of customer emotions you would like to analyze (video or audio calls, voicemail, etc.). 

You can then use this information to further develop or adjust business areas, such as your commercial proposition, customer experience, and brand awareness.

Excited to see how AI could create an impact for you?

Contact our Solution Lead to learn more about sentiment analysis and what it could do for your organization.