How I built a fine-tuned AI-Powered Sentiment Dashboard with Azure, Power Automate and Power BI
A case study for building an automated system that analyzes and understands the pain points of customers.
Overview
In the lottery, gambling, and betting industry, customer feedback is very valuable. If our customers have any trouble with our products or services, they can easily switch to another company. To understand our customers' pain points, we check our social media channels, websites, and inboxes daily. Product managers want to learn about each issue promptly, and if an issue is fixable, they want to address it immediately because it can reduce the revenue and impact customer happiness.
The Challenge
We use a tool for social media listening, but it doesn't completely support our local Azerbaijani language. As a result, it often shows the sentiment of each sentence incorrectly. This confusion means our product managers can't understand the issues promptly.
For example, the tool might show negative feedback, but when we check, we see it's actually positive because a customer used slang that the tool doesn't understand. Alongside this, the tool also confuses our company's name with those of foreign companies. We receive more than 3,000 pieces of feedback per month, but when we analyze them, we find that only about half are actually related to our brand.
The Approach
I selected 400 pieces of feedback and labeled them manually: 100 for each sentiment (positive, neutral, and negative), and 100 for unrelated comments.
I trained an Azerbaijani language model with this small dataset. After each test, I checked the results manually and retrained the model to increase its accuracy.
I used Power Automate (both Desktop and Cloud) to gather data from the tool and add it to Azure Storage.
I used Azure Functions to automate tasks whenever new data was added to Storage. The Function sent this data to the model for analysis and then saved the results back into storage.
Finally, I created a powerful and intuitive dashboard in Power BI to visualize the insights easily.
Key Findings
Approximately 50-60% of customer feedback with our keywords related with brand.
We have negative Net Sentiment Score. Because in reality our brand's negative feedback is more than positive.
We know customer main pain points about each project related topics.
The Solution
First, I manually labeled 400 pieces of feedback. I used 80% of this dataset to train the model and the remaining 20% to test it. After manually checking the output, I retrained the model repeatedly with a larger dataset, finally achieving an 85% accuracy level.
I used the same technique for topic and project analysis. This is because, besides sentiment, we also wanted to learn which specific topics and projects were causing customer pain points. I used the same training approach for both of these models.
After that, I created an automated flow using both Power Automate Desktop and Cloud. I used the Desktop version for Chrome-based operations like entering the social media tool's website, checking the date range, and uploading the data. The Cloud version was used for adding this data to Azure Storage.
Next, I used Azure Functions to trigger a process whenever the dataset in Azure Storage was updated. The function would send the new data to the model for analysis and then add the resulting output back to Storage.
The final step was creating the Power BI dashboard. I built a visually appealing dashboard with custom cards and charts, and I used DAX for calculations. To make the dashboard interactive, I used SVGs combined with DAX code. As a result, when a stakeholder changes the date range, the visuals update dynamically.
Impact & Results
The measurable outcomes that made a difference
Product managers can now see the pain points and issues promptly.
We started to develop our products to provide a better service.
We learned why some customers were switching to another company.
We run informative campaigns for our customers.
Tools & Technologies
The technology stack that powered this project: