Challenge
In the age of social media, the ability to quickly triage and handle a large volume of incoming customer enquiries is crucial for any large organisation.
Our client asked us to help them to recognise and classify emotive content into categories such as “happy”, “angry” and “worry” in order to help them to prioritise incoming customer enquiries and prevent escalating complaints and customer churn.
Further complicating things, the incoming enquiries were in the Austrian dialect of German.
Solution
Filament initially investigated whether any existing “software-as-a-service” emotion classification APIs such as those provided by IBM, Microsoft and Google were appropriate for the task. We found that most of the APIs did not support Austrian german and eventually opted to train our own model.
A small number of utterances were provided by the client to validate our model. However, a large amount of annotated data was required to train a new model. We worked closely with the client and a German crowd annotation company to get over 10,000 examples of emotive customer enquires annotated and used best practices for obtaining inter-annotator-agreement in order to make sure that our training data was consistent.
We then trained a recurrent-neural model to encode the customer enquiries word-by-word and generate an emotion output. We were able to obtain 75% accuracy across 4 classes of emotion.
We worked with the client to productionise and deploy our emotion model as a REST API that they could integrate with their chat platform in order to handle emotion detection and triage in live customer chat sessions.
The initial data study was completed over 8 weeks
The production system was built over 24 weeks.