Customer Feedback Analysis for Financial Services LLM

Project Overview:

Objective

The goal was to build a comprehensive dataset of customer feedback, annotated for sentiment and topic, to improve the accuracy of LLMs in analyzing sentiment and categorizing customer support issues in financial services.

Additionally, this dataset was designed to enhance AI-driven customer feedback analysis, enabling financial institutions to better understand customer behavior, improve service quality, and automate support processes.

Scope

The dataset includes both structure and unstructure customer feedback from financial service providers, including banks and insurance companies. The feedback was annotate for sentiment and categorize by topic to ensure precise analysis by LLMs.

This approach helps LLMs process real-world financial interactions, improving their ability to generate accurate insights and support decision-making.

Sources

Customer Feedback Collection: The data was sourced from 30,000 feedback entries provided by customers of various financial services, including both banks and insurance firms.

Data Collection Metrics

Total Feedback Entries: 30,000 feedback entries were collected.
Sentiment Tags: Each feedback entry was annotated with 3 sentiment tags (positive, negative, or neutral).

These annotations enable AI models to identify customer sentiment patterns and prioritize critical issues more effectively.

Annotation Process

Stages

Sentiment Classification: Annotators classified the feedback into three categories: positive, negative, or neutral.
Topic Categorization: Feedback was also categorized by topics relevant to financial services, such as account management, loan services, and customer support.

Annotation Metrics

Total Sentiment Annotations: 30,000 sentiment tags were applied.
Team Involvement: 35 annotators worked on the project over a duration of 1 month.

This large-scale annotation process ensures high-quality training data for LLM-based financial applications.

Quality Assurance

Stages

Annotation Accuracy: Continuous checks were performed to ensure that sentiment and topic annotations were accurate and aligned with the feedback content.
Consistency Checks: Regular reviews were conducted to maintain consistent tagging across all entries.

QA Metrics

Sentiment Accuracy: The project achieved a high accuracy rate in correctly identifying customer sentiment across feedback entries.

This improves the effectiveness of AI-powered customer support systems.

Topic Classification Accuracy: The feedback was accurately categorized by topic, enhancing the relevance of responses generated by LLMs.

Accurate topic classification enables faster issue resolution and better customer experience.

Conclusion

The creation of this dataset marked a significant improvement in the ability of LLMs to analyze customer sentiment and support issues within the financial services industry. By accurately interpreting customer feedback, the dataset has contributed to better automated customer service responses and increased customer satisfaction.